Oct 11 10:26:06.925295 master-1 systemd[1]: Starting Kubernetes Kubelet... Oct 11 10:26:12.087624 master-1 kubenswrapper[4771]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 11 10:26:12.087624 master-1 kubenswrapper[4771]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 11 10:26:12.087624 master-1 kubenswrapper[4771]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 11 10:26:12.087624 master-1 kubenswrapper[4771]: Flag --register-with-taints has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 11 10:26:12.087624 master-1 kubenswrapper[4771]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI. Oct 11 10:26:12.087624 master-1 kubenswrapper[4771]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 11 10:26:12.090281 master-1 kubenswrapper[4771]: I1011 10:26:12.088539 4771 server.go:211] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Oct 11 10:26:12.093343 master-1 kubenswrapper[4771]: W1011 10:26:12.093280 4771 feature_gate.go:330] unrecognized feature gate: VolumeGroupSnapshot Oct 11 10:26:12.093343 master-1 kubenswrapper[4771]: W1011 10:26:12.093318 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallAzure Oct 11 10:26:12.093343 master-1 kubenswrapper[4771]: W1011 10:26:12.093327 4771 feature_gate.go:330] unrecognized feature gate: DNSNameResolver Oct 11 10:26:12.093343 master-1 kubenswrapper[4771]: W1011 10:26:12.093335 4771 feature_gate.go:330] unrecognized feature gate: RouteAdvertisements Oct 11 10:26:12.093343 master-1 kubenswrapper[4771]: W1011 10:26:12.093344 4771 feature_gate.go:330] unrecognized feature gate: NetworkDiagnosticsConfig Oct 11 10:26:12.093343 master-1 kubenswrapper[4771]: W1011 10:26:12.093381 4771 feature_gate.go:351] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093393 4771 feature_gate.go:330] unrecognized feature gate: VSphereDriverConfiguration Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093402 4771 feature_gate.go:330] unrecognized feature gate: PrivateHostedZoneAWS Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093411 4771 feature_gate.go:330] unrecognized feature gate: CSIDriverSharedResource Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093419 4771 feature_gate.go:330] unrecognized feature gate: NutanixMultiSubnets Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093427 4771 feature_gate.go:330] unrecognized feature gate: NetworkSegmentation Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093435 4771 feature_gate.go:330] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093443 4771 feature_gate.go:330] unrecognized feature gate: ManagedBootImages Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093450 4771 feature_gate.go:330] unrecognized feature gate: ChunkSizeMiB Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093458 4771 feature_gate.go:330] unrecognized feature gate: AWSClusterHostedDNS Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093466 4771 feature_gate.go:330] unrecognized feature gate: MixedCPUsAllocation Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093474 4771 feature_gate.go:330] unrecognized feature gate: NetworkLiveMigration Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093490 4771 feature_gate.go:330] unrecognized feature gate: SignatureStores Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093498 4771 feature_gate.go:330] unrecognized feature gate: ImageStreamImportMode Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093506 4771 feature_gate.go:330] unrecognized feature gate: InsightsConfig Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093514 4771 feature_gate.go:330] unrecognized feature gate: GatewayAPI Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093522 4771 feature_gate.go:330] unrecognized feature gate: AdditionalRoutingCapabilities Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093529 4771 feature_gate.go:330] unrecognized feature gate: AlibabaPlatform Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093540 4771 feature_gate.go:353] Setting GA feature gate DisableKubeletCloudCredentialProviders=true. It will be removed in a future release. Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093551 4771 feature_gate.go:330] unrecognized feature gate: VSphereControlPlaneMachineSet Oct 11 10:26:12.093829 master-1 kubenswrapper[4771]: W1011 10:26:12.093560 4771 feature_gate.go:330] unrecognized feature gate: ExternalOIDC Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093569 4771 feature_gate.go:330] unrecognized feature gate: AutomatedEtcdBackup Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093577 4771 feature_gate.go:330] unrecognized feature gate: PlatformOperators Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093585 4771 feature_gate.go:330] unrecognized feature gate: ManagedBootImagesAWS Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093593 4771 feature_gate.go:330] unrecognized feature gate: NodeDisruptionPolicy Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093600 4771 feature_gate.go:330] unrecognized feature gate: InsightsConfigAPI Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093608 4771 feature_gate.go:330] unrecognized feature gate: Example Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093618 4771 feature_gate.go:330] unrecognized feature gate: MinimumKubeletVersion Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093625 4771 feature_gate.go:330] unrecognized feature gate: ClusterMonitoringConfig Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093635 4771 feature_gate.go:353] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release. Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093646 4771 feature_gate.go:330] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093655 4771 feature_gate.go:330] unrecognized feature gate: PinnedImages Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093664 4771 feature_gate.go:330] unrecognized feature gate: ClusterAPIInstall Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093673 4771 feature_gate.go:330] unrecognized feature gate: VSphereMultiVCenters Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093689 4771 feature_gate.go:330] unrecognized feature gate: SetEIPForNLBIngressController Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093698 4771 feature_gate.go:330] unrecognized feature gate: UpgradeStatus Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093707 4771 feature_gate.go:330] unrecognized feature gate: PersistentIPsForVirtualization Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093715 4771 feature_gate.go:330] unrecognized feature gate: AWSEFSDriverVolumeMetrics Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093722 4771 feature_gate.go:330] unrecognized feature gate: GCPLabelsTags Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093731 4771 feature_gate.go:330] unrecognized feature gate: IngressControllerLBSubnetsAWS Oct 11 10:26:12.095477 master-1 kubenswrapper[4771]: W1011 10:26:12.093739 4771 feature_gate.go:330] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093747 4771 feature_gate.go:330] unrecognized feature gate: OVNObservability Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093755 4771 feature_gate.go:330] unrecognized feature gate: EtcdBackendQuota Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093763 4771 feature_gate.go:330] unrecognized feature gate: AzureWorkloadIdentity Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093771 4771 feature_gate.go:330] unrecognized feature gate: MetricsCollectionProfiles Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093778 4771 feature_gate.go:330] unrecognized feature gate: InsightsRuntimeExtractor Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093786 4771 feature_gate.go:330] unrecognized feature gate: MachineConfigNodes Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093794 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIProviderOpenStack Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093801 4771 feature_gate.go:330] unrecognized feature gate: SigstoreImageVerification Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093809 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093816 4771 feature_gate.go:330] unrecognized feature gate: AdminNetworkPolicy Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093824 4771 feature_gate.go:330] unrecognized feature gate: VSphereStaticIPs Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093833 4771 feature_gate.go:330] unrecognized feature gate: BootcNodeManagement Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093845 4771 feature_gate.go:353] Setting GA feature gate ValidatingAdmissionPolicy=true. It will be removed in a future release. Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093855 4771 feature_gate.go:330] unrecognized feature gate: BuildCSIVolumes Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093864 4771 feature_gate.go:330] unrecognized feature gate: BareMetalLoadBalancer Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093873 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallGCP Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093882 4771 feature_gate.go:330] unrecognized feature gate: InsightsOnDemandDataGather Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093891 4771 feature_gate.go:330] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 11 10:26:12.096828 master-1 kubenswrapper[4771]: W1011 10:26:12.093899 4771 feature_gate.go:330] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: W1011 10:26:12.093908 4771 feature_gate.go:330] unrecognized feature gate: GCPClusterHostedDNS Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: W1011 10:26:12.093916 4771 feature_gate.go:330] unrecognized feature gate: HardwareSpeed Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: W1011 10:26:12.093925 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIMigration Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: W1011 10:26:12.093933 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallAWS Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: W1011 10:26:12.093941 4771 feature_gate.go:330] unrecognized feature gate: OnClusterBuild Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: W1011 10:26:12.093948 4771 feature_gate.go:330] unrecognized feature gate: NewOLM Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: W1011 10:26:12.093962 4771 feature_gate.go:330] unrecognized feature gate: VSphereMultiNetworks Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094117 4771 flags.go:64] FLAG: --address="0.0.0.0" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094137 4771 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094151 4771 flags.go:64] FLAG: --anonymous-auth="true" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094162 4771 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094173 4771 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094182 4771 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094194 4771 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094205 4771 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094214 4771 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094223 4771 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094232 4771 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094242 4771 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094252 4771 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094261 4771 flags.go:64] FLAG: --cgroup-root="" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094269 4771 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 11 10:26:12.098100 master-1 kubenswrapper[4771]: I1011 10:26:12.094278 4771 flags.go:64] FLAG: --client-ca-file="" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094287 4771 flags.go:64] FLAG: --cloud-config="" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094295 4771 flags.go:64] FLAG: --cloud-provider="" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094304 4771 flags.go:64] FLAG: --cluster-dns="[]" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094317 4771 flags.go:64] FLAG: --cluster-domain="" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094325 4771 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094334 4771 flags.go:64] FLAG: --config-dir="" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094344 4771 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094400 4771 flags.go:64] FLAG: --container-log-max-files="5" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094412 4771 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094421 4771 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094429 4771 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094440 4771 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094449 4771 flags.go:64] FLAG: --contention-profiling="false" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094458 4771 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094467 4771 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094481 4771 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094490 4771 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094502 4771 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094511 4771 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094520 4771 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094530 4771 flags.go:64] FLAG: --enable-load-reader="false" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094539 4771 flags.go:64] FLAG: --enable-server="true" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094550 4771 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094564 4771 flags.go:64] FLAG: --event-burst="100" Oct 11 10:26:12.099657 master-1 kubenswrapper[4771]: I1011 10:26:12.094576 4771 flags.go:64] FLAG: --event-qps="50" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094587 4771 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094599 4771 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094611 4771 flags.go:64] FLAG: --eviction-hard="" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094625 4771 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094637 4771 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094648 4771 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094659 4771 flags.go:64] FLAG: --eviction-soft="" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094668 4771 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094678 4771 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094687 4771 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094696 4771 flags.go:64] FLAG: --experimental-mounter-path="" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094704 4771 flags.go:64] FLAG: --fail-cgroupv1="false" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094714 4771 flags.go:64] FLAG: --fail-swap-on="true" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094722 4771 flags.go:64] FLAG: --feature-gates="" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094733 4771 flags.go:64] FLAG: --file-check-frequency="20s" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094742 4771 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094752 4771 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094761 4771 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094770 4771 flags.go:64] FLAG: --healthz-port="10248" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094780 4771 flags.go:64] FLAG: --help="false" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094789 4771 flags.go:64] FLAG: --hostname-override="" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094797 4771 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094809 4771 flags.go:64] FLAG: --http-check-frequency="20s" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094818 4771 flags.go:64] FLAG: --image-credential-provider-bin-dir="" Oct 11 10:26:12.101287 master-1 kubenswrapper[4771]: I1011 10:26:12.094827 4771 flags.go:64] FLAG: --image-credential-provider-config="" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.094836 4771 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.094844 4771 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.094853 4771 flags.go:64] FLAG: --image-service-endpoint="" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.094862 4771 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.094871 4771 flags.go:64] FLAG: --kube-api-burst="100" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.094880 4771 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.094891 4771 flags.go:64] FLAG: --kube-api-qps="50" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095006 4771 flags.go:64] FLAG: --kube-reserved="" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095021 4771 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095033 4771 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095044 4771 flags.go:64] FLAG: --kubelet-cgroups="" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095055 4771 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095067 4771 flags.go:64] FLAG: --lock-file="" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095078 4771 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095090 4771 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095102 4771 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095118 4771 flags.go:64] FLAG: --log-json-split-stream="false" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095129 4771 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095140 4771 flags.go:64] FLAG: --log-text-split-stream="false" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095151 4771 flags.go:64] FLAG: --logging-format="text" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095161 4771 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095170 4771 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095179 4771 flags.go:64] FLAG: --manifest-url="" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095188 4771 flags.go:64] FLAG: --manifest-url-header="" Oct 11 10:26:12.102800 master-1 kubenswrapper[4771]: I1011 10:26:12.095200 4771 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095209 4771 flags.go:64] FLAG: --max-open-files="1000000" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095220 4771 flags.go:64] FLAG: --max-pods="110" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095229 4771 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095238 4771 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095251 4771 flags.go:64] FLAG: --memory-manager-policy="None" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095260 4771 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095269 4771 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095278 4771 flags.go:64] FLAG: --node-ip="192.168.34.11" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095287 4771 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.openshift.io/os_id=rhcos" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095309 4771 flags.go:64] FLAG: --node-status-max-images="50" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095320 4771 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095331 4771 flags.go:64] FLAG: --oom-score-adj="-999" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095342 4771 flags.go:64] FLAG: --pod-cidr="" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095416 4771 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2d66b9dbe1d071d7372c477a78835fb65b48ea82db00d23e9086af5cfcb194ad" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095434 4771 flags.go:64] FLAG: --pod-manifest-path="" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095443 4771 flags.go:64] FLAG: --pod-max-pids="-1" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095452 4771 flags.go:64] FLAG: --pods-per-core="0" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095463 4771 flags.go:64] FLAG: --port="10250" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095472 4771 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095482 4771 flags.go:64] FLAG: --provider-id="" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095492 4771 flags.go:64] FLAG: --qos-reserved="" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095502 4771 flags.go:64] FLAG: --read-only-port="10255" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095514 4771 flags.go:64] FLAG: --register-node="true" Oct 11 10:26:12.104348 master-1 kubenswrapper[4771]: I1011 10:26:12.095523 4771 flags.go:64] FLAG: --register-schedulable="true" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095533 4771 flags.go:64] FLAG: --register-with-taints="node-role.kubernetes.io/master=:NoSchedule" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095549 4771 flags.go:64] FLAG: --registry-burst="10" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095558 4771 flags.go:64] FLAG: --registry-qps="5" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095566 4771 flags.go:64] FLAG: --reserved-cpus="" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095575 4771 flags.go:64] FLAG: --reserved-memory="" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095586 4771 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095595 4771 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095604 4771 flags.go:64] FLAG: --rotate-certificates="false" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095613 4771 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095622 4771 flags.go:64] FLAG: --runonce="false" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095630 4771 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095640 4771 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095649 4771 flags.go:64] FLAG: --seccomp-default="false" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095661 4771 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095670 4771 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095679 4771 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095689 4771 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095698 4771 flags.go:64] FLAG: --storage-driver-password="root" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095706 4771 flags.go:64] FLAG: --storage-driver-secure="false" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095715 4771 flags.go:64] FLAG: --storage-driver-table="stats" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095723 4771 flags.go:64] FLAG: --storage-driver-user="root" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095732 4771 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095741 4771 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095750 4771 flags.go:64] FLAG: --system-cgroups="" Oct 11 10:26:12.105822 master-1 kubenswrapper[4771]: I1011 10:26:12.095759 4771 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095773 4771 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095781 4771 flags.go:64] FLAG: --tls-cert-file="" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095791 4771 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095803 4771 flags.go:64] FLAG: --tls-min-version="" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095813 4771 flags.go:64] FLAG: --tls-private-key-file="" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095822 4771 flags.go:64] FLAG: --topology-manager-policy="none" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095831 4771 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095840 4771 flags.go:64] FLAG: --topology-manager-scope="container" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095849 4771 flags.go:64] FLAG: --v="2" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095860 4771 flags.go:64] FLAG: --version="false" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095871 4771 flags.go:64] FLAG: --vmodule="" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095881 4771 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: I1011 10:26:12.095890 4771 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096100 4771 feature_gate.go:330] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096111 4771 feature_gate.go:330] unrecognized feature gate: VSphereControlPlaneMachineSet Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096120 4771 feature_gate.go:330] unrecognized feature gate: CSIDriverSharedResource Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096129 4771 feature_gate.go:330] unrecognized feature gate: VSphereMultiVCenters Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096139 4771 feature_gate.go:330] unrecognized feature gate: AWSEFSDriverVolumeMetrics Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096147 4771 feature_gate.go:330] unrecognized feature gate: UpgradeStatus Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096155 4771 feature_gate.go:330] unrecognized feature gate: PersistentIPsForVirtualization Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096166 4771 feature_gate.go:330] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096173 4771 feature_gate.go:330] unrecognized feature gate: InsightsConfigAPI Oct 11 10:26:12.107204 master-1 kubenswrapper[4771]: W1011 10:26:12.096181 4771 feature_gate.go:330] unrecognized feature gate: PrivateHostedZoneAWS Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096189 4771 feature_gate.go:330] unrecognized feature gate: MixedCPUsAllocation Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096196 4771 feature_gate.go:330] unrecognized feature gate: BootcNodeManagement Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096204 4771 feature_gate.go:330] unrecognized feature gate: AWSClusterHostedDNS Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096212 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIMigration Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096219 4771 feature_gate.go:330] unrecognized feature gate: ManagedBootImagesAWS Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096227 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallGCP Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096235 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallAWS Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096243 4771 feature_gate.go:330] unrecognized feature gate: ClusterAPIInstall Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096250 4771 feature_gate.go:330] unrecognized feature gate: VolumeGroupSnapshot Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096258 4771 feature_gate.go:330] unrecognized feature gate: AdminNetworkPolicy Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096266 4771 feature_gate.go:330] unrecognized feature gate: GCPClusterHostedDNS Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096274 4771 feature_gate.go:330] unrecognized feature gate: VSphereDriverConfiguration Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096282 4771 feature_gate.go:330] unrecognized feature gate: InsightsRuntimeExtractor Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096289 4771 feature_gate.go:330] unrecognized feature gate: InsightsOnDemandDataGather Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096297 4771 feature_gate.go:330] unrecognized feature gate: SignatureStores Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096305 4771 feature_gate.go:330] unrecognized feature gate: SetEIPForNLBIngressController Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096312 4771 feature_gate.go:330] unrecognized feature gate: AzureWorkloadIdentity Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096321 4771 feature_gate.go:330] unrecognized feature gate: ImageStreamImportMode Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096329 4771 feature_gate.go:330] unrecognized feature gate: GCPLabelsTags Oct 11 10:26:12.108553 master-1 kubenswrapper[4771]: W1011 10:26:12.096336 4771 feature_gate.go:330] unrecognized feature gate: GatewayAPI Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096345 4771 feature_gate.go:330] unrecognized feature gate: SigstoreImageVerification Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096377 4771 feature_gate.go:330] unrecognized feature gate: ClusterMonitoringConfig Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096385 4771 feature_gate.go:330] unrecognized feature gate: MachineConfigNodes Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096392 4771 feature_gate.go:330] unrecognized feature gate: OnClusterBuild Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096400 4771 feature_gate.go:330] unrecognized feature gate: ExternalOIDC Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096408 4771 feature_gate.go:330] unrecognized feature gate: AdditionalRoutingCapabilities Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096415 4771 feature_gate.go:330] unrecognized feature gate: VSphereStaticIPs Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096423 4771 feature_gate.go:330] unrecognized feature gate: HardwareSpeed Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096431 4771 feature_gate.go:330] unrecognized feature gate: PinnedImages Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096444 4771 feature_gate.go:330] unrecognized feature gate: NodeDisruptionPolicy Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096451 4771 feature_gate.go:330] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096459 4771 feature_gate.go:330] unrecognized feature gate: ManagedBootImages Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096467 4771 feature_gate.go:330] unrecognized feature gate: VSphereMultiNetworks Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096474 4771 feature_gate.go:330] unrecognized feature gate: RouteAdvertisements Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096482 4771 feature_gate.go:330] unrecognized feature gate: NetworkLiveMigration Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096490 4771 feature_gate.go:330] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096499 4771 feature_gate.go:330] unrecognized feature gate: IngressControllerLBSubnetsAWS Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096507 4771 feature_gate.go:330] unrecognized feature gate: EtcdBackendQuota Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096515 4771 feature_gate.go:330] unrecognized feature gate: BareMetalLoadBalancer Oct 11 10:26:12.109968 master-1 kubenswrapper[4771]: W1011 10:26:12.096523 4771 feature_gate.go:330] unrecognized feature gate: DNSNameResolver Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096533 4771 feature_gate.go:353] Setting GA feature gate ValidatingAdmissionPolicy=true. It will be removed in a future release. Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096547 4771 feature_gate.go:330] unrecognized feature gate: Example Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096555 4771 feature_gate.go:330] unrecognized feature gate: NetworkDiagnosticsConfig Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096565 4771 feature_gate.go:353] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release. Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096575 4771 feature_gate.go:330] unrecognized feature gate: NetworkSegmentation Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096583 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096592 4771 feature_gate.go:330] unrecognized feature gate: MetricsCollectionProfiles Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096600 4771 feature_gate.go:330] unrecognized feature gate: PlatformOperators Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096608 4771 feature_gate.go:330] unrecognized feature gate: AutomatedEtcdBackup Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096615 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallAzure Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096626 4771 feature_gate.go:353] Setting GA feature gate DisableKubeletCloudCredentialProviders=true. It will be removed in a future release. Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096636 4771 feature_gate.go:330] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096646 4771 feature_gate.go:330] unrecognized feature gate: OVNObservability Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096655 4771 feature_gate.go:330] unrecognized feature gate: MinimumKubeletVersion Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096664 4771 feature_gate.go:330] unrecognized feature gate: ChunkSizeMiB Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096674 4771 feature_gate.go:351] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096683 4771 feature_gate.go:330] unrecognized feature gate: InsightsConfig Oct 11 10:26:12.111158 master-1 kubenswrapper[4771]: W1011 10:26:12.096693 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIProviderOpenStack Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.096701 4771 feature_gate.go:330] unrecognized feature gate: AlibabaPlatform Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.096710 4771 feature_gate.go:330] unrecognized feature gate: NutanixMultiSubnets Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.096718 4771 feature_gate.go:330] unrecognized feature gate: NewOLM Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.096732 4771 feature_gate.go:330] unrecognized feature gate: BuildCSIVolumes Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: I1011 10:26:12.096745 4771 feature_gate.go:386] feature gates: {map[CloudDualStackNodeIPs:true DisableKubeletCloudCredentialProviders:true DynamicResourceAllocation:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false NodeSwap:false ProcMountType:false RouteExternalCertificate:false ServiceAccountTokenNodeBinding:false TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:false UserNamespacesSupport:false ValidatingAdmissionPolicy:true VolumeAttributesClass:false]} Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: I1011 10:26:12.111075 4771 server.go:491] "Kubelet version" kubeletVersion="v1.31.13" Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: I1011 10:26:12.111107 4771 server.go:493] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.111228 4771 feature_gate.go:330] unrecognized feature gate: AWSEFSDriverVolumeMetrics Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.111240 4771 feature_gate.go:330] unrecognized feature gate: GCPLabelsTags Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.111250 4771 feature_gate.go:330] unrecognized feature gate: AWSClusterHostedDNS Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.111259 4771 feature_gate.go:330] unrecognized feature gate: ExternalOIDC Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.111270 4771 feature_gate.go:353] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release. Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.111280 4771 feature_gate.go:330] unrecognized feature gate: HardwareSpeed Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.111290 4771 feature_gate.go:330] unrecognized feature gate: BootcNodeManagement Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.111299 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallAWS Oct 11 10:26:12.112172 master-1 kubenswrapper[4771]: W1011 10:26:12.111308 4771 feature_gate.go:330] unrecognized feature gate: UpgradeStatus Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111316 4771 feature_gate.go:330] unrecognized feature gate: BuildCSIVolumes Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111324 4771 feature_gate.go:330] unrecognized feature gate: OnClusterBuild Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111331 4771 feature_gate.go:330] unrecognized feature gate: AdditionalRoutingCapabilities Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111339 4771 feature_gate.go:330] unrecognized feature gate: NetworkSegmentation Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111347 4771 feature_gate.go:330] unrecognized feature gate: SetEIPForNLBIngressController Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111389 4771 feature_gate.go:330] unrecognized feature gate: PersistentIPsForVirtualization Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111400 4771 feature_gate.go:330] unrecognized feature gate: SigstoreImageVerification Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111413 4771 feature_gate.go:353] Setting GA feature gate ValidatingAdmissionPolicy=true. It will be removed in a future release. Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111427 4771 feature_gate.go:330] unrecognized feature gate: InsightsOnDemandDataGather Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111435 4771 feature_gate.go:330] unrecognized feature gate: DNSNameResolver Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111444 4771 feature_gate.go:330] unrecognized feature gate: RouteAdvertisements Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111453 4771 feature_gate.go:330] unrecognized feature gate: NetworkLiveMigration Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111462 4771 feature_gate.go:330] unrecognized feature gate: InsightsConfig Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111473 4771 feature_gate.go:353] Setting GA feature gate DisableKubeletCloudCredentialProviders=true. It will be removed in a future release. Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111482 4771 feature_gate.go:330] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111492 4771 feature_gate.go:330] unrecognized feature gate: AzureWorkloadIdentity Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111501 4771 feature_gate.go:330] unrecognized feature gate: MixedCPUsAllocation Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111511 4771 feature_gate.go:330] unrecognized feature gate: ClusterAPIInstall Oct 11 10:26:12.113164 master-1 kubenswrapper[4771]: W1011 10:26:12.111518 4771 feature_gate.go:330] unrecognized feature gate: ChunkSizeMiB Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111527 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallAzure Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111535 4771 feature_gate.go:330] unrecognized feature gate: ClusterMonitoringConfig Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111543 4771 feature_gate.go:330] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111551 4771 feature_gate.go:330] unrecognized feature gate: VSphereMultiVCenters Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111559 4771 feature_gate.go:330] unrecognized feature gate: OVNObservability Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111567 4771 feature_gate.go:330] unrecognized feature gate: EtcdBackendQuota Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111579 4771 feature_gate.go:351] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111588 4771 feature_gate.go:330] unrecognized feature gate: ManagedBootImagesAWS Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111596 4771 feature_gate.go:330] unrecognized feature gate: Example Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111604 4771 feature_gate.go:330] unrecognized feature gate: NewOLM Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111612 4771 feature_gate.go:330] unrecognized feature gate: NetworkDiagnosticsConfig Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111620 4771 feature_gate.go:330] unrecognized feature gate: VSphereControlPlaneMachineSet Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111629 4771 feature_gate.go:330] unrecognized feature gate: PinnedImages Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111637 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIProviderOpenStack Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111645 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111653 4771 feature_gate.go:330] unrecognized feature gate: PrivateHostedZoneAWS Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111660 4771 feature_gate.go:330] unrecognized feature gate: MachineConfigNodes Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111668 4771 feature_gate.go:330] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111676 4771 feature_gate.go:330] unrecognized feature gate: CSIDriverSharedResource Oct 11 10:26:12.114431 master-1 kubenswrapper[4771]: W1011 10:26:12.111683 4771 feature_gate.go:330] unrecognized feature gate: GatewayAPI Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111691 4771 feature_gate.go:330] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111699 4771 feature_gate.go:330] unrecognized feature gate: VSphereDriverConfiguration Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111707 4771 feature_gate.go:330] unrecognized feature gate: ManagedBootImages Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111714 4771 feature_gate.go:330] unrecognized feature gate: PlatformOperators Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111722 4771 feature_gate.go:330] unrecognized feature gate: MinimumKubeletVersion Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111730 4771 feature_gate.go:330] unrecognized feature gate: MetricsCollectionProfiles Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111738 4771 feature_gate.go:330] unrecognized feature gate: AdminNetworkPolicy Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111746 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallGCP Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111753 4771 feature_gate.go:330] unrecognized feature gate: InsightsConfigAPI Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111761 4771 feature_gate.go:330] unrecognized feature gate: AutomatedEtcdBackup Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111769 4771 feature_gate.go:330] unrecognized feature gate: InsightsRuntimeExtractor Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111777 4771 feature_gate.go:330] unrecognized feature gate: BareMetalLoadBalancer Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111784 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIMigration Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111792 4771 feature_gate.go:330] unrecognized feature gate: VolumeGroupSnapshot Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111800 4771 feature_gate.go:330] unrecognized feature gate: SignatureStores Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111811 4771 feature_gate.go:330] unrecognized feature gate: ImageStreamImportMode Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111820 4771 feature_gate.go:330] unrecognized feature gate: AlibabaPlatform Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111828 4771 feature_gate.go:330] unrecognized feature gate: NodeDisruptionPolicy Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111837 4771 feature_gate.go:330] unrecognized feature gate: IngressControllerLBSubnetsAWS Oct 11 10:26:12.115947 master-1 kubenswrapper[4771]: W1011 10:26:12.111845 4771 feature_gate.go:330] unrecognized feature gate: VSphereStaticIPs Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.111852 4771 feature_gate.go:330] unrecognized feature gate: GCPClusterHostedDNS Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.111861 4771 feature_gate.go:330] unrecognized feature gate: VSphereMultiNetworks Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.111877 4771 feature_gate.go:330] unrecognized feature gate: NutanixMultiSubnets Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.111886 4771 feature_gate.go:330] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: I1011 10:26:12.111898 4771 feature_gate.go:386] feature gates: {map[CloudDualStackNodeIPs:true DisableKubeletCloudCredentialProviders:true DynamicResourceAllocation:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false NodeSwap:false ProcMountType:false RouteExternalCertificate:false ServiceAccountTokenNodeBinding:false TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:false UserNamespacesSupport:false ValidatingAdmissionPolicy:true VolumeAttributesClass:false]} Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112112 4771 feature_gate.go:330] unrecognized feature gate: MachineConfigNodes Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112126 4771 feature_gate.go:330] unrecognized feature gate: ManagedBootImagesAWS Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112135 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112145 4771 feature_gate.go:330] unrecognized feature gate: AutomatedEtcdBackup Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112154 4771 feature_gate.go:330] unrecognized feature gate: BuildCSIVolumes Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112164 4771 feature_gate.go:330] unrecognized feature gate: AlibabaPlatform Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112173 4771 feature_gate.go:330] unrecognized feature gate: NetworkSegmentation Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112182 4771 feature_gate.go:330] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112191 4771 feature_gate.go:330] unrecognized feature gate: InsightsConfig Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112199 4771 feature_gate.go:330] unrecognized feature gate: DNSNameResolver Oct 11 10:26:12.117538 master-1 kubenswrapper[4771]: W1011 10:26:12.112207 4771 feature_gate.go:330] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112216 4771 feature_gate.go:330] unrecognized feature gate: AdditionalRoutingCapabilities Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112225 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallGCP Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112233 4771 feature_gate.go:330] unrecognized feature gate: CSIDriverSharedResource Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112241 4771 feature_gate.go:330] unrecognized feature gate: AdminNetworkPolicy Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112249 4771 feature_gate.go:330] unrecognized feature gate: AWSClusterHostedDNS Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112256 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallAWS Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112265 4771 feature_gate.go:330] unrecognized feature gate: VolumeGroupSnapshot Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112272 4771 feature_gate.go:330] unrecognized feature gate: GCPClusterHostedDNS Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112280 4771 feature_gate.go:330] unrecognized feature gate: ExternalOIDC Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112288 4771 feature_gate.go:330] unrecognized feature gate: MetricsCollectionProfiles Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112296 4771 feature_gate.go:330] unrecognized feature gate: EtcdBackendQuota Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112305 4771 feature_gate.go:330] unrecognized feature gate: VSphereMultiNetworks Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112313 4771 feature_gate.go:330] unrecognized feature gate: AzureWorkloadIdentity Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112321 4771 feature_gate.go:330] unrecognized feature gate: GatewayAPI Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112328 4771 feature_gate.go:330] unrecognized feature gate: BareMetalLoadBalancer Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112336 4771 feature_gate.go:330] unrecognized feature gate: PinnedImages Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112344 4771 feature_gate.go:330] unrecognized feature gate: OnClusterBuild Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112379 4771 feature_gate.go:330] unrecognized feature gate: AWSEFSDriverVolumeMetrics Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112391 4771 feature_gate.go:330] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 11 10:26:12.118804 master-1 kubenswrapper[4771]: W1011 10:26:12.112402 4771 feature_gate.go:330] unrecognized feature gate: HardwareSpeed Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112411 4771 feature_gate.go:330] unrecognized feature gate: InsightsRuntimeExtractor Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112421 4771 feature_gate.go:330] unrecognized feature gate: ManagedBootImages Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112433 4771 feature_gate.go:330] unrecognized feature gate: VSphereStaticIPs Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112443 4771 feature_gate.go:330] unrecognized feature gate: BootcNodeManagement Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112451 4771 feature_gate.go:330] unrecognized feature gate: SigstoreImageVerification Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112459 4771 feature_gate.go:330] unrecognized feature gate: UpgradeStatus Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112467 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIProviderOpenStack Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112475 4771 feature_gate.go:330] unrecognized feature gate: NewOLM Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112482 4771 feature_gate.go:330] unrecognized feature gate: Example Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112490 4771 feature_gate.go:330] unrecognized feature gate: MachineAPIMigration Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112498 4771 feature_gate.go:330] unrecognized feature gate: PrivateHostedZoneAWS Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112506 4771 feature_gate.go:330] unrecognized feature gate: RouteAdvertisements Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112514 4771 feature_gate.go:330] unrecognized feature gate: InsightsConfigAPI Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112522 4771 feature_gate.go:330] unrecognized feature gate: SetEIPForNLBIngressController Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112530 4771 feature_gate.go:330] unrecognized feature gate: ImageStreamImportMode Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112537 4771 feature_gate.go:330] unrecognized feature gate: VSphereControlPlaneMachineSet Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112547 4771 feature_gate.go:353] Setting GA feature gate DisableKubeletCloudCredentialProviders=true. It will be removed in a future release. Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112557 4771 feature_gate.go:353] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release. Oct 11 10:26:12.120614 master-1 kubenswrapper[4771]: W1011 10:26:12.112567 4771 feature_gate.go:330] unrecognized feature gate: NetworkDiagnosticsConfig Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112576 4771 feature_gate.go:330] unrecognized feature gate: ClusterMonitoringConfig Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112585 4771 feature_gate.go:330] unrecognized feature gate: MultiArchInstallAzure Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112593 4771 feature_gate.go:330] unrecognized feature gate: MixedCPUsAllocation Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112604 4771 feature_gate.go:353] Setting GA feature gate ValidatingAdmissionPolicy=true. It will be removed in a future release. Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112616 4771 feature_gate.go:330] unrecognized feature gate: NetworkLiveMigration Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112625 4771 feature_gate.go:330] unrecognized feature gate: OVNObservability Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112633 4771 feature_gate.go:330] unrecognized feature gate: IngressControllerLBSubnetsAWS Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112641 4771 feature_gate.go:330] unrecognized feature gate: VSphereMultiVCenters Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112649 4771 feature_gate.go:330] unrecognized feature gate: ChunkSizeMiB Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112660 4771 feature_gate.go:351] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112670 4771 feature_gate.go:330] unrecognized feature gate: NodeDisruptionPolicy Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112679 4771 feature_gate.go:330] unrecognized feature gate: ClusterAPIInstall Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112687 4771 feature_gate.go:330] unrecognized feature gate: SignatureStores Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112696 4771 feature_gate.go:330] unrecognized feature gate: MinimumKubeletVersion Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112705 4771 feature_gate.go:330] unrecognized feature gate: NutanixMultiSubnets Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112713 4771 feature_gate.go:330] unrecognized feature gate: GCPLabelsTags Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112721 4771 feature_gate.go:330] unrecognized feature gate: PlatformOperators Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112730 4771 feature_gate.go:330] unrecognized feature gate: PersistentIPsForVirtualization Oct 11 10:26:12.121666 master-1 kubenswrapper[4771]: W1011 10:26:12.112738 4771 feature_gate.go:330] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 11 10:26:12.122713 master-1 kubenswrapper[4771]: W1011 10:26:12.112747 4771 feature_gate.go:330] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 11 10:26:12.122713 master-1 kubenswrapper[4771]: W1011 10:26:12.112757 4771 feature_gate.go:330] unrecognized feature gate: InsightsOnDemandDataGather Oct 11 10:26:12.122713 master-1 kubenswrapper[4771]: W1011 10:26:12.112765 4771 feature_gate.go:330] unrecognized feature gate: VSphereDriverConfiguration Oct 11 10:26:12.122713 master-1 kubenswrapper[4771]: I1011 10:26:12.112776 4771 feature_gate.go:386] feature gates: {map[CloudDualStackNodeIPs:true DisableKubeletCloudCredentialProviders:true DynamicResourceAllocation:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false NodeSwap:false ProcMountType:false RouteExternalCertificate:false ServiceAccountTokenNodeBinding:false TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:false UserNamespacesSupport:false ValidatingAdmissionPolicy:true VolumeAttributesClass:false]} Oct 11 10:26:12.122713 master-1 kubenswrapper[4771]: I1011 10:26:12.113001 4771 server.go:940] "Client rotation is on, will bootstrap in background" Oct 11 10:26:12.122713 master-1 kubenswrapper[4771]: I1011 10:26:12.118815 4771 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 11 10:26:12.122713 master-1 kubenswrapper[4771]: I1011 10:26:12.120723 4771 server.go:997] "Starting client certificate rotation" Oct 11 10:26:12.122713 master-1 kubenswrapper[4771]: I1011 10:26:12.120748 4771 certificate_manager.go:356] kubernetes.io/kube-apiserver-client-kubelet: Certificate rotation is enabled Oct 11 10:26:12.122713 master-1 kubenswrapper[4771]: I1011 10:26:12.121028 4771 certificate_manager.go:356] kubernetes.io/kube-apiserver-client-kubelet: Rotating certificates Oct 11 10:26:12.152954 master-1 kubenswrapper[4771]: I1011 10:26:12.152827 4771 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 11 10:26:12.157066 master-1 kubenswrapper[4771]: I1011 10:26:12.156889 4771 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 11 10:26:12.175403 master-1 kubenswrapper[4771]: I1011 10:26:12.175300 4771 log.go:25] "Validated CRI v1 runtime API" Oct 11 10:26:12.184188 master-1 kubenswrapper[4771]: I1011 10:26:12.184125 4771 log.go:25] "Validated CRI v1 image API" Oct 11 10:26:12.186840 master-1 kubenswrapper[4771]: I1011 10:26:12.186789 4771 server.go:1437] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 11 10:26:12.191398 master-1 kubenswrapper[4771]: I1011 10:26:12.191252 4771 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/vda2 910678ff-f77e-4a7d-8d53-86f2ac47a823:/dev/vda4 e0ffcf34-f743-4501-a23b-0da71751fe05:/dev/vda3] Oct 11 10:26:12.191398 master-1 kubenswrapper[4771]: I1011 10:26:12.191296 4771 fs.go:136] Filesystem partitions: map[/dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /dev/vda3:{mountpoint:/boot major:252 minor:3 fsType:ext4 blockSize:0} /dev/vda4:{mountpoint:/var major:252 minor:4 fsType:xfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:30 fsType:tmpfs blockSize:0}] Oct 11 10:26:12.195537 master-1 kubenswrapper[4771]: I1011 10:26:12.195488 4771 reflector.go:368] Caches populated for *v1.CertificateSigningRequest from k8s.io/client-go/tools/watch/informerwatcher.go:146 Oct 11 10:26:12.216254 master-1 kubenswrapper[4771]: I1011 10:26:12.215714 4771 manager.go:217] Machine: {Timestamp:2025-10-11 10:26:12.213629119 +0000 UTC m=+4.187855630 CPUVendorID:AuthenticAMD NumCores:16 NumPhysicalCores:1 NumSockets:16 CpuFrequency:2800000 MemoryCapacity:50514157568 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:fe2953088e5c4cf684aae6b361d97c24 SystemUUID:fe295308-8e5c-4cf6-84aa-e6b361d97c24 BootID:2a34dbce-856d-4b83-8fb9-526c6edd4eae Filesystems:[{Device:/dev/vda4 DeviceMajor:252 DeviceMinor:4 Capacity:214143315968 Type:vfs Inodes:104594880 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:30 Capacity:25257078784 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/vda3 DeviceMajor:252 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:25257078784 Type:vfs Inodes:6166279 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:10102833152 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[252:0:{Name:vda Major:252 Minor:0 Size:214748364800 Scheduler:none} 252:16:{Name:vdb Major:252 Minor:16 Size:21474836480 Scheduler:none} 252:32:{Name:vdc Major:252 Minor:32 Size:21474836480 Scheduler:none} 252:48:{Name:vdd Major:252 Minor:48 Size:21474836480 Scheduler:none} 252:64:{Name:vde Major:252 Minor:64 Size:21474836480 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:fa:16:3e:3e:b4:b2 Speed:0 Mtu:9000} {Name:eth0 MacAddress:fa:16:3e:3e:b4:b2 Speed:-1 Mtu:9000} {Name:eth1 MacAddress:fa:16:3e:c8:69:ef Speed:-1 Mtu:9000} {Name:eth2 MacAddress:fa:16:3e:98:96:aa Speed:-1 Mtu:9000} {Name:ovs-system MacAddress:6a:ea:32:ab:c4:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:50514157568 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:0 Size:16777216 Type:Unified Level:3}] SocketID:0 BookID: DrawerID:} {Id:0 Threads:[1] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:1 Size:16777216 Type:Unified Level:3}] SocketID:1 BookID: DrawerID:} {Id:0 Threads:[10] Caches:[{Id:10 Size:32768 Type:Data Level:1} {Id:10 Size:32768 Type:Instruction Level:1} {Id:10 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:10 Size:16777216 Type:Unified Level:3}] SocketID:10 BookID: DrawerID:} {Id:0 Threads:[11] Caches:[{Id:11 Size:32768 Type:Data Level:1} {Id:11 Size:32768 Type:Instruction Level:1} {Id:11 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:11 Size:16777216 Type:Unified Level:3}] SocketID:11 BookID: DrawerID:} {Id:0 Threads:[12] Caches:[{Id:12 Size:32768 Type:Data Level:1} {Id:12 Size:32768 Type:Instruction Level:1} {Id:12 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:12 Size:16777216 Type:Unified Level:3}] SocketID:12 BookID: DrawerID:} {Id:0 Threads:[13] Caches:[{Id:13 Size:32768 Type:Data Level:1} {Id:13 Size:32768 Type:Instruction Level:1} {Id:13 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:13 Size:16777216 Type:Unified Level:3}] SocketID:13 BookID: DrawerID:} {Id:0 Threads:[14] Caches:[{Id:14 Size:32768 Type:Data Level:1} {Id:14 Size:32768 Type:Instruction Level:1} {Id:14 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:14 Size:16777216 Type:Unified Level:3}] SocketID:14 BookID: DrawerID:} {Id:0 Threads:[15] Caches:[{Id:15 Size:32768 Type:Data Level:1} {Id:15 Size:32768 Type:Instruction Level:1} {Id:15 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:15 Size:16777216 Type:Unified Level:3}] SocketID:15 BookID: DrawerID:} {Id:0 Threads:[2] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:2 Size:16777216 Type:Unified Level:3}] SocketID:2 BookID: DrawerID:} {Id:0 Threads:[3] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:3 Size:16777216 Type:Unified Level:3}] SocketID:3 BookID: DrawerID:} {Id:0 Threads:[4] Caches:[{Id:4 Size:32768 Type:Data Level:1} {Id:4 Size:32768 Type:Instruction Level:1} {Id:4 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:4 Size:16777216 Type:Unified Level:3}] SocketID:4 BookID: DrawerID:} {Id:0 Threads:[5] Caches:[{Id:5 Size:32768 Type:Data Level:1} {Id:5 Size:32768 Type:Instruction Level:1} {Id:5 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:5 Size:16777216 Type:Unified Level:3}] SocketID:5 BookID: DrawerID:} {Id:0 Threads:[6] Caches:[{Id:6 Size:32768 Type:Data Level:1} {Id:6 Size:32768 Type:Instruction Level:1} {Id:6 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:6 Size:16777216 Type:Unified Level:3}] SocketID:6 BookID: DrawerID:} {Id:0 Threads:[7] Caches:[{Id:7 Size:32768 Type:Data Level:1} {Id:7 Size:32768 Type:Instruction Level:1} {Id:7 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:7 Size:16777216 Type:Unified Level:3}] SocketID:7 BookID: DrawerID:} {Id:0 Threads:[8] Caches:[{Id:8 Size:32768 Type:Data Level:1} {Id:8 Size:32768 Type:Instruction Level:1} {Id:8 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:8 Size:16777216 Type:Unified Level:3}] SocketID:8 BookID: DrawerID:} {Id:0 Threads:[9] Caches:[{Id:9 Size:32768 Type:Data Level:1} {Id:9 Size:32768 Type:Instruction Level:1} {Id:9 Size:524288 Type:Unified Level:2}] UncoreCaches:[{Id:9 Size:16777216 Type:Unified Level:3}] SocketID:9 BookID: DrawerID:}] Caches:[] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 11 10:26:12.216254 master-1 kubenswrapper[4771]: I1011 10:26:12.216175 4771 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 11 10:26:12.216640 master-1 kubenswrapper[4771]: I1011 10:26:12.216432 4771 manager.go:233] Version: {KernelVersion:5.14.0-427.91.1.el9_4.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 418.94.202509241235-0 DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 11 10:26:12.216977 master-1 kubenswrapper[4771]: I1011 10:26:12.216925 4771 swap_util.go:113] "Swap is on" /proc/swaps contents="Filename\t\t\t\tType\t\tSize\t\tUsed\t\tPriority" Oct 11 10:26:12.217345 master-1 kubenswrapper[4771]: I1011 10:26:12.217273 4771 container_manager_linux.go:267] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 11 10:26:12.217735 master-1 kubenswrapper[4771]: I1011 10:26:12.217330 4771 container_manager_linux.go:272] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"master-1","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"ExperimentalMemoryManagerPolicy":"None","ExperimentalMemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 11 10:26:12.217735 master-1 kubenswrapper[4771]: I1011 10:26:12.217728 4771 topology_manager.go:138] "Creating topology manager with none policy" Oct 11 10:26:12.217947 master-1 kubenswrapper[4771]: I1011 10:26:12.217749 4771 container_manager_linux.go:303] "Creating device plugin manager" Oct 11 10:26:12.217947 master-1 kubenswrapper[4771]: I1011 10:26:12.217779 4771 manager.go:142] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 11 10:26:12.217947 master-1 kubenswrapper[4771]: I1011 10:26:12.217808 4771 server.go:66] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 11 10:26:12.219231 master-1 kubenswrapper[4771]: I1011 10:26:12.219174 4771 state_mem.go:36] "Initialized new in-memory state store" Oct 11 10:26:12.219348 master-1 kubenswrapper[4771]: I1011 10:26:12.219329 4771 server.go:1245] "Using root directory" path="/var/lib/kubelet" Oct 11 10:26:12.225488 master-1 kubenswrapper[4771]: I1011 10:26:12.225443 4771 kubelet.go:418] "Attempting to sync node with API server" Oct 11 10:26:12.225488 master-1 kubenswrapper[4771]: I1011 10:26:12.225480 4771 kubelet.go:313] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 11 10:26:12.225709 master-1 kubenswrapper[4771]: I1011 10:26:12.225511 4771 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 11 10:26:12.225709 master-1 kubenswrapper[4771]: I1011 10:26:12.225546 4771 kubelet.go:324] "Adding apiserver pod source" Oct 11 10:26:12.226268 master-1 kubenswrapper[4771]: I1011 10:26:12.226175 4771 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 11 10:26:12.236766 master-1 kubenswrapper[4771]: I1011 10:26:12.236691 4771 kuberuntime_manager.go:262] "Container runtime initialized" containerRuntime="cri-o" version="1.31.12-3.rhaos4.18.gitdc59c78.el9" apiVersion="v1" Oct 11 10:26:12.238768 master-1 kubenswrapper[4771]: I1011 10:26:12.238729 4771 kubelet.go:854] "Not starting ClusterTrustBundle informer because we are in static kubelet mode" Oct 11 10:26:12.239993 master-1 kubenswrapper[4771]: I1011 10:26:12.239952 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 11 10:26:12.240060 master-1 kubenswrapper[4771]: I1011 10:26:12.239996 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 11 10:26:12.240060 master-1 kubenswrapper[4771]: I1011 10:26:12.240014 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 11 10:26:12.240060 master-1 kubenswrapper[4771]: I1011 10:26:12.240029 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 11 10:26:12.240060 master-1 kubenswrapper[4771]: I1011 10:26:12.240051 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 11 10:26:12.240189 master-1 kubenswrapper[4771]: I1011 10:26:12.240065 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 11 10:26:12.240189 master-1 kubenswrapper[4771]: I1011 10:26:12.240080 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 11 10:26:12.240189 master-1 kubenswrapper[4771]: I1011 10:26:12.240105 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 11 10:26:12.240189 master-1 kubenswrapper[4771]: I1011 10:26:12.240121 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 11 10:26:12.240189 master-1 kubenswrapper[4771]: I1011 10:26:12.240135 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 11 10:26:12.240189 master-1 kubenswrapper[4771]: I1011 10:26:12.240155 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 11 10:26:12.240782 master-1 kubenswrapper[4771]: I1011 10:26:12.240743 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 11 10:26:12.243099 master-1 kubenswrapper[4771]: I1011 10:26:12.243063 4771 plugins.go:603] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 11 10:26:12.244113 master-1 kubenswrapper[4771]: W1011 10:26:12.244074 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Node: nodes "master-1" is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope Oct 11 10:26:12.244184 master-1 kubenswrapper[4771]: I1011 10:26:12.244122 4771 server.go:1280] "Started kubelet" Oct 11 10:26:12.244184 master-1 kubenswrapper[4771]: E1011 10:26:12.244166 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Node: failed to list *v1.Node: nodes \"master-1\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:12.244653 master-1 kubenswrapper[4771]: W1011 10:26:12.244577 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope Oct 11 10:26:12.244734 master-1 kubenswrapper[4771]: E1011 10:26:12.244673 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:12.245253 master-1 kubenswrapper[4771]: I1011 10:26:12.245092 4771 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 11 10:26:12.245448 master-1 kubenswrapper[4771]: I1011 10:26:12.245275 4771 server_v1.go:47] "podresources" method="list" useActivePods=true Oct 11 10:26:12.246389 master-1 systemd[1]: Started Kubernetes Kubelet. Oct 11 10:26:12.247639 master-1 kubenswrapper[4771]: I1011 10:26:12.247585 4771 server.go:236] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 11 10:26:12.247702 master-1 kubenswrapper[4771]: I1011 10:26:12.247578 4771 server.go:163] "Starting to listen" address="0.0.0.0" port=10250 Oct 11 10:26:12.248914 master-1 kubenswrapper[4771]: I1011 10:26:12.248865 4771 certificate_manager.go:356] kubernetes.io/kubelet-serving: Certificate rotation is enabled Oct 11 10:26:12.248963 master-1 kubenswrapper[4771]: I1011 10:26:12.248944 4771 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Oct 11 10:26:12.249226 master-1 kubenswrapper[4771]: I1011 10:26:12.249187 4771 volume_manager.go:287] "The desired_state_of_world populator starts" Oct 11 10:26:12.249226 master-1 kubenswrapper[4771]: I1011 10:26:12.249223 4771 volume_manager.go:289] "Starting Kubelet Volume Manager" Oct 11 10:26:12.249318 master-1 kubenswrapper[4771]: I1011 10:26:12.249273 4771 desired_state_of_world_populator.go:147] "Desired state populator starts to run" Oct 11 10:26:12.249318 master-1 kubenswrapper[4771]: E1011 10:26:12.249274 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:12.249493 master-1 kubenswrapper[4771]: I1011 10:26:12.249444 4771 reconstruct.go:97] "Volume reconstruction finished" Oct 11 10:26:12.249493 master-1 kubenswrapper[4771]: I1011 10:26:12.249484 4771 reconciler.go:26] "Reconciler: start to sync state" Oct 11 10:26:12.250129 master-1 kubenswrapper[4771]: I1011 10:26:12.250091 4771 server.go:449] "Adding debug handlers to kubelet server" Oct 11 10:26:12.255397 master-1 kubenswrapper[4771]: I1011 10:26:12.255257 4771 csi_plugin.go:884] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "master-1" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 11 10:26:12.255724 master-1 kubenswrapper[4771]: E1011 10:26:12.255453 4771 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"master-1\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 11 10:26:12.255931 master-1 kubenswrapper[4771]: W1011 10:26:12.255879 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope Oct 11 10:26:12.256152 master-1 kubenswrapper[4771]: E1011 10:26:12.255934 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:12.256152 master-1 kubenswrapper[4771]: E1011 10:26:12.254886 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75aeefbfb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.244077563 +0000 UTC m=+4.218304044,LastTimestamp:2025-10-11 10:26:12.244077563 +0000 UTC m=+4.218304044,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.259167 master-1 kubenswrapper[4771]: I1011 10:26:12.258791 4771 factory.go:55] Registering systemd factory Oct 11 10:26:12.259167 master-1 kubenswrapper[4771]: I1011 10:26:12.258823 4771 factory.go:221] Registration of the systemd container factory successfully Oct 11 10:26:12.259622 master-1 kubenswrapper[4771]: I1011 10:26:12.259332 4771 factory.go:153] Registering CRI-O factory Oct 11 10:26:12.259622 master-1 kubenswrapper[4771]: I1011 10:26:12.259399 4771 factory.go:221] Registration of the crio container factory successfully Oct 11 10:26:12.259622 master-1 kubenswrapper[4771]: I1011 10:26:12.259518 4771 factory.go:219] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 11 10:26:12.259622 master-1 kubenswrapper[4771]: I1011 10:26:12.259568 4771 factory.go:103] Registering Raw factory Oct 11 10:26:12.259622 master-1 kubenswrapper[4771]: I1011 10:26:12.259595 4771 manager.go:1196] Started watching for new ooms in manager Oct 11 10:26:12.260885 master-1 kubenswrapper[4771]: I1011 10:26:12.260830 4771 manager.go:319] Starting recovery of all containers Oct 11 10:26:12.265303 master-1 kubenswrapper[4771]: E1011 10:26:12.265254 4771 kubelet.go:1495] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 11 10:26:12.286460 master-1 kubenswrapper[4771]: I1011 10:26:12.286046 4771 manager.go:324] Recovery completed Oct 11 10:26:12.300631 master-1 kubenswrapper[4771]: I1011 10:26:12.300573 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:12.303066 master-1 kubenswrapper[4771]: I1011 10:26:12.302956 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:12.303066 master-1 kubenswrapper[4771]: I1011 10:26:12.303038 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:12.303066 master-1 kubenswrapper[4771]: I1011 10:26:12.303051 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:12.304683 master-1 kubenswrapper[4771]: I1011 10:26:12.304652 4771 cpu_manager.go:225] "Starting CPU manager" policy="none" Oct 11 10:26:12.304683 master-1 kubenswrapper[4771]: I1011 10:26:12.304674 4771 cpu_manager.go:226] "Reconciling" reconcilePeriod="10s" Oct 11 10:26:12.304876 master-1 kubenswrapper[4771]: I1011 10:26:12.304716 4771 state_mem.go:36] "Initialized new in-memory state store" Oct 11 10:26:12.308100 master-1 kubenswrapper[4771]: E1011 10:26:12.307910 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e726697 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node master-1 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303021719 +0000 UTC m=+4.277248170,LastTimestamp:2025-10-11 10:26:12.303021719 +0000 UTC m=+4.277248170,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.319487 master-1 kubenswrapper[4771]: E1011 10:26:12.319247 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72c42f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node master-1 status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303045679 +0000 UTC m=+4.277272130,LastTimestamp:2025-10-11 10:26:12.303045679 +0000 UTC m=+4.277272130,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.329083 master-1 kubenswrapper[4771]: E1011 10:26:12.328884 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72f48b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node master-1 status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303058059 +0000 UTC m=+4.277284510,LastTimestamp:2025-10-11 10:26:12.303058059 +0000 UTC m=+4.277284510,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.341246 master-1 kubenswrapper[4771]: I1011 10:26:12.341054 4771 policy_none.go:49] "None policy: Start" Oct 11 10:26:12.343013 master-1 kubenswrapper[4771]: I1011 10:26:12.342975 4771 memory_manager.go:170] "Starting memorymanager" policy="None" Oct 11 10:26:12.343141 master-1 kubenswrapper[4771]: I1011 10:26:12.343043 4771 state_mem.go:35] "Initializing new in-memory state store" Oct 11 10:26:12.350407 master-1 kubenswrapper[4771]: E1011 10:26:12.350341 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:12.426453 master-1 kubenswrapper[4771]: I1011 10:26:12.426347 4771 manager.go:334] "Starting Device Plugin manager" Oct 11 10:26:12.426677 master-1 kubenswrapper[4771]: I1011 10:26:12.426495 4771 manager.go:513] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found" Oct 11 10:26:12.426677 master-1 kubenswrapper[4771]: I1011 10:26:12.426527 4771 server.go:79] "Starting device plugin registration server" Oct 11 10:26:12.427285 master-1 kubenswrapper[4771]: I1011 10:26:12.427234 4771 eviction_manager.go:189] "Eviction manager: starting control loop" Oct 11 10:26:12.428279 master-1 kubenswrapper[4771]: I1011 10:26:12.427339 4771 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 11 10:26:12.428484 master-1 kubenswrapper[4771]: I1011 10:26:12.428344 4771 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 11 10:26:12.428562 master-1 kubenswrapper[4771]: I1011 10:26:12.428519 4771 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Oct 11 10:26:12.428562 master-1 kubenswrapper[4771]: I1011 10:26:12.428536 4771 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Oct 11 10:26:12.429760 master-1 kubenswrapper[4771]: E1011 10:26:12.429722 4771 eviction_manager.go:285] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"master-1\" not found" Oct 11 10:26:12.431905 master-1 kubenswrapper[4771]: I1011 10:26:12.431818 4771 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv4" Oct 11 10:26:12.435587 master-1 kubenswrapper[4771]: I1011 10:26:12.435530 4771 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv6" Oct 11 10:26:12.435703 master-1 kubenswrapper[4771]: I1011 10:26:12.435616 4771 status_manager.go:217] "Starting to sync pod status with apiserver" Oct 11 10:26:12.435703 master-1 kubenswrapper[4771]: I1011 10:26:12.435657 4771 kubelet.go:2335] "Starting kubelet main sync loop" Oct 11 10:26:12.435910 master-1 kubenswrapper[4771]: E1011 10:26:12.435797 4771 kubelet.go:2359] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 11 10:26:12.440635 master-1 kubenswrapper[4771]: W1011 10:26:12.440556 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User "system:anonymous" cannot list resource "runtimeclasses" in API group "node.k8s.io" at the cluster scope Oct 11 10:26:12.440635 master-1 kubenswrapper[4771]: E1011 10:26:12.440608 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:12.441098 master-1 kubenswrapper[4771]: E1011 10:26:12.440951 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e7660e895e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.43069475 +0000 UTC m=+4.404921221,LastTimestamp:2025-10-11 10:26:12.43069475 +0000 UTC m=+4.404921221,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.458252 master-1 kubenswrapper[4771]: E1011 10:26:12.458168 4771 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"master-1\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Oct 11 10:26:12.528590 master-1 kubenswrapper[4771]: I1011 10:26:12.528422 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:12.529923 master-1 kubenswrapper[4771]: I1011 10:26:12.529866 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:12.529923 master-1 kubenswrapper[4771]: I1011 10:26:12.529920 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:12.530130 master-1 kubenswrapper[4771]: I1011 10:26:12.529959 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:12.530130 master-1 kubenswrapper[4771]: I1011 10:26:12.530056 4771 kubelet_node_status.go:76] "Attempting to register node" node="master-1" Oct 11 10:26:12.536864 master-1 kubenswrapper[4771]: I1011 10:26:12.536798 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-master-1"] Oct 11 10:26:12.536986 master-1 kubenswrapper[4771]: I1011 10:26:12.536928 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:12.538123 master-1 kubenswrapper[4771]: I1011 10:26:12.537953 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:12.538123 master-1 kubenswrapper[4771]: I1011 10:26:12.538013 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:12.538123 master-1 kubenswrapper[4771]: I1011 10:26:12.538031 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:12.538572 master-1 kubenswrapper[4771]: I1011 10:26:12.538498 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" Oct 11 10:26:12.538572 master-1 kubenswrapper[4771]: I1011 10:26:12.538547 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:12.539990 master-1 kubenswrapper[4771]: I1011 10:26:12.539884 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:12.540244 master-1 kubenswrapper[4771]: I1011 10:26:12.540021 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:12.540244 master-1 kubenswrapper[4771]: I1011 10:26:12.540042 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:12.541811 master-1 kubenswrapper[4771]: E1011 10:26:12.541753 4771 kubelet_node_status.go:99] "Unable to register node with API server" err="nodes is forbidden: User \"system:anonymous\" cannot create resource \"nodes\" in API group \"\" at the cluster scope" node="master-1" Oct 11 10:26:12.541811 master-1 kubenswrapper[4771]: E1011 10:26:12.541612 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e726697\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e726697 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node master-1 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303021719 +0000 UTC m=+4.277248170,LastTimestamp:2025-10-11 10:26:12.529905392 +0000 UTC m=+4.504131873,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.555090 master-1 kubenswrapper[4771]: E1011 10:26:12.554761 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72c42f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72c42f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node master-1 status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303045679 +0000 UTC m=+4.277272130,LastTimestamp:2025-10-11 10:26:12.529932693 +0000 UTC m=+4.504159174,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.567991 master-1 kubenswrapper[4771]: E1011 10:26:12.567812 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72f48b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72f48b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node master-1 status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303058059 +0000 UTC m=+4.277284510,LastTimestamp:2025-10-11 10:26:12.529969584 +0000 UTC m=+4.504196065,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.577877 master-1 kubenswrapper[4771]: E1011 10:26:12.577724 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e726697\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e726697 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node master-1 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303021719 +0000 UTC m=+4.277248170,LastTimestamp:2025-10-11 10:26:12.537992016 +0000 UTC m=+4.512218497,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.587989 master-1 kubenswrapper[4771]: E1011 10:26:12.587755 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72c42f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72c42f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node master-1 status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303045679 +0000 UTC m=+4.277272130,LastTimestamp:2025-10-11 10:26:12.538024586 +0000 UTC m=+4.512251067,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.598519 master-1 kubenswrapper[4771]: E1011 10:26:12.598255 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72f48b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72f48b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node master-1 status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303058059 +0000 UTC m=+4.277284510,LastTimestamp:2025-10-11 10:26:12.538040577 +0000 UTC m=+4.512267058,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.608909 master-1 kubenswrapper[4771]: E1011 10:26:12.608755 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e726697\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e726697 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node master-1 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303021719 +0000 UTC m=+4.277248170,LastTimestamp:2025-10-11 10:26:12.539924047 +0000 UTC m=+4.514150518,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.618482 master-1 kubenswrapper[4771]: E1011 10:26:12.618230 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72c42f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72c42f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node master-1 status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303045679 +0000 UTC m=+4.277272130,LastTimestamp:2025-10-11 10:26:12.54003546 +0000 UTC m=+4.514261941,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.628695 master-1 kubenswrapper[4771]: E1011 10:26:12.628481 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72f48b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72f48b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node master-1 status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303058059 +0000 UTC m=+4.277284510,LastTimestamp:2025-10-11 10:26:12.54005138 +0000 UTC m=+4.514277851,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.651098 master-1 kubenswrapper[4771]: I1011 10:26:12.651026 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3273b5dc02e0d8cacbf64fe78c713d50-etc-kube\") pod \"kube-rbac-proxy-crio-master-1\" (UID: \"3273b5dc02e0d8cacbf64fe78c713d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" Oct 11 10:26:12.651098 master-1 kubenswrapper[4771]: I1011 10:26:12.651093 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3273b5dc02e0d8cacbf64fe78c713d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-master-1\" (UID: \"3273b5dc02e0d8cacbf64fe78c713d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" Oct 11 10:26:12.743140 master-1 kubenswrapper[4771]: I1011 10:26:12.743041 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:12.744915 master-1 kubenswrapper[4771]: I1011 10:26:12.744865 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:12.745018 master-1 kubenswrapper[4771]: I1011 10:26:12.744956 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:12.745018 master-1 kubenswrapper[4771]: I1011 10:26:12.744976 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:12.745129 master-1 kubenswrapper[4771]: I1011 10:26:12.745056 4771 kubelet_node_status.go:76] "Attempting to register node" node="master-1" Oct 11 10:26:12.752230 master-1 kubenswrapper[4771]: I1011 10:26:12.752152 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3273b5dc02e0d8cacbf64fe78c713d50-etc-kube\") pod \"kube-rbac-proxy-crio-master-1\" (UID: \"3273b5dc02e0d8cacbf64fe78c713d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" Oct 11 10:26:12.752430 master-1 kubenswrapper[4771]: I1011 10:26:12.752245 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3273b5dc02e0d8cacbf64fe78c713d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-master-1\" (UID: \"3273b5dc02e0d8cacbf64fe78c713d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" Oct 11 10:26:12.752430 master-1 kubenswrapper[4771]: I1011 10:26:12.752342 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3273b5dc02e0d8cacbf64fe78c713d50-etc-kube\") pod \"kube-rbac-proxy-crio-master-1\" (UID: \"3273b5dc02e0d8cacbf64fe78c713d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" Oct 11 10:26:12.752519 master-1 kubenswrapper[4771]: I1011 10:26:12.752422 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3273b5dc02e0d8cacbf64fe78c713d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-master-1\" (UID: \"3273b5dc02e0d8cacbf64fe78c713d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" Oct 11 10:26:12.759747 master-1 kubenswrapper[4771]: E1011 10:26:12.759595 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e726697\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e726697 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node master-1 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303021719 +0000 UTC m=+4.277248170,LastTimestamp:2025-10-11 10:26:12.744934318 +0000 UTC m=+4.719160789,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.759998 master-1 kubenswrapper[4771]: E1011 10:26:12.759891 4771 kubelet_node_status.go:99] "Unable to register node with API server" err="nodes is forbidden: User \"system:anonymous\" cannot create resource \"nodes\" in API group \"\" at the cluster scope" node="master-1" Oct 11 10:26:12.769944 master-1 kubenswrapper[4771]: E1011 10:26:12.769804 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72c42f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72c42f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node master-1 status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303045679 +0000 UTC m=+4.277272130,LastTimestamp:2025-10-11 10:26:12.744969049 +0000 UTC m=+4.719195530,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.779961 master-1 kubenswrapper[4771]: E1011 10:26:12.779704 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72f48b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72f48b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node master-1 status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303058059 +0000 UTC m=+4.277284510,LastTimestamp:2025-10-11 10:26:12.744985559 +0000 UTC m=+4.719212040,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:12.870930 master-1 kubenswrapper[4771]: E1011 10:26:12.870699 4771 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"master-1\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Oct 11 10:26:12.875715 master-1 kubenswrapper[4771]: I1011 10:26:12.875638 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" Oct 11 10:26:13.076859 master-1 kubenswrapper[4771]: W1011 10:26:13.076760 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope Oct 11 10:26:13.076859 master-1 kubenswrapper[4771]: E1011 10:26:13.076849 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:13.160986 master-1 kubenswrapper[4771]: I1011 10:26:13.160722 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:13.162739 master-1 kubenswrapper[4771]: I1011 10:26:13.162211 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:13.162739 master-1 kubenswrapper[4771]: I1011 10:26:13.162240 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:13.162739 master-1 kubenswrapper[4771]: I1011 10:26:13.162252 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:13.162739 master-1 kubenswrapper[4771]: I1011 10:26:13.162297 4771 kubelet_node_status.go:76] "Attempting to register node" node="master-1" Oct 11 10:26:13.174034 master-1 kubenswrapper[4771]: E1011 10:26:13.173868 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e726697\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e726697 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node master-1 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303021719 +0000 UTC m=+4.277248170,LastTimestamp:2025-10-11 10:26:13.162229626 +0000 UTC m=+5.136456077,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:13.174291 master-1 kubenswrapper[4771]: E1011 10:26:13.174227 4771 kubelet_node_status.go:99] "Unable to register node with API server" err="nodes is forbidden: User \"system:anonymous\" cannot create resource \"nodes\" in API group \"\" at the cluster scope" node="master-1" Oct 11 10:26:13.183514 master-1 kubenswrapper[4771]: E1011 10:26:13.183285 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72c42f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72c42f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node master-1 status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303045679 +0000 UTC m=+4.277272130,LastTimestamp:2025-10-11 10:26:13.162246686 +0000 UTC m=+5.136473137,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:13.193291 master-1 kubenswrapper[4771]: E1011 10:26:13.193140 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72f48b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72f48b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node master-1 status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303058059 +0000 UTC m=+4.277284510,LastTimestamp:2025-10-11 10:26:13.162261717 +0000 UTC m=+5.136488168,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:13.266880 master-1 kubenswrapper[4771]: I1011 10:26:13.266750 4771 csi_plugin.go:884] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "master-1" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 11 10:26:13.363452 master-1 kubenswrapper[4771]: W1011 10:26:13.363235 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Node: nodes "master-1" is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope Oct 11 10:26:13.363452 master-1 kubenswrapper[4771]: E1011 10:26:13.363327 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Node: failed to list *v1.Node: nodes \"master-1\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:13.472490 master-1 kubenswrapper[4771]: W1011 10:26:13.472222 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope Oct 11 10:26:13.472490 master-1 kubenswrapper[4771]: E1011 10:26:13.472313 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:13.539079 master-1 kubenswrapper[4771]: W1011 10:26:13.538950 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User "system:anonymous" cannot list resource "runtimeclasses" in API group "node.k8s.io" at the cluster scope Oct 11 10:26:13.539079 master-1 kubenswrapper[4771]: E1011 10:26:13.539045 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:13.680409 master-1 kubenswrapper[4771]: E1011 10:26:13.680281 4771 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"master-1\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Oct 11 10:26:13.718577 master-1 kubenswrapper[4771]: W1011 10:26:13.718435 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3273b5dc02e0d8cacbf64fe78c713d50.slice/crio-d3beeb8f52049cfbfd643d5240049be7138d9ff83afdc0d99012e206f07dfb78 WatchSource:0}: Error finding container d3beeb8f52049cfbfd643d5240049be7138d9ff83afdc0d99012e206f07dfb78: Status 404 returned error can't find the container with id d3beeb8f52049cfbfd643d5240049be7138d9ff83afdc0d99012e206f07dfb78 Oct 11 10:26:13.722035 master-1 kubenswrapper[4771]: I1011 10:26:13.721986 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 10:26:13.733996 master-1 kubenswrapper[4771]: E1011 10:26:13.733782 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-master-1.186d68e7b304f330 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-master-1,UID:3273b5dc02e0d8cacbf64fe78c713d50,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f22b65e5c744a32d3955dd7c36d809e3114a8aa501b44c00330dfda886c21169\",Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:13.721912112 +0000 UTC m=+5.696138553,LastTimestamp:2025-10-11 10:26:13.721912112 +0000 UTC m=+5.696138553,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:13.974864 master-1 kubenswrapper[4771]: I1011 10:26:13.974608 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:13.976600 master-1 kubenswrapper[4771]: I1011 10:26:13.976510 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:13.976600 master-1 kubenswrapper[4771]: I1011 10:26:13.976567 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:13.976600 master-1 kubenswrapper[4771]: I1011 10:26:13.976586 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:13.976852 master-1 kubenswrapper[4771]: I1011 10:26:13.976632 4771 kubelet_node_status.go:76] "Attempting to register node" node="master-1" Oct 11 10:26:13.987376 master-1 kubenswrapper[4771]: E1011 10:26:13.987272 4771 kubelet_node_status.go:99] "Unable to register node with API server" err="nodes is forbidden: User \"system:anonymous\" cannot create resource \"nodes\" in API group \"\" at the cluster scope" node="master-1" Oct 11 10:26:13.988041 master-1 kubenswrapper[4771]: E1011 10:26:13.987915 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e726697\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e726697 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node master-1 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303021719 +0000 UTC m=+4.277248170,LastTimestamp:2025-10-11 10:26:13.97654609 +0000 UTC m=+5.950772561,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:13.998063 master-1 kubenswrapper[4771]: E1011 10:26:13.997886 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72c42f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72c42f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node master-1 status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303045679 +0000 UTC m=+4.277272130,LastTimestamp:2025-10-11 10:26:13.976577261 +0000 UTC m=+5.950803742,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:14.009232 master-1 kubenswrapper[4771]: E1011 10:26:14.009044 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72f48b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72f48b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node master-1 status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303058059 +0000 UTC m=+4.277284510,LastTimestamp:2025-10-11 10:26:13.976595591 +0000 UTC m=+5.950822062,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:14.266752 master-1 kubenswrapper[4771]: I1011 10:26:14.266539 4771 csi_plugin.go:884] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "master-1" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 11 10:26:14.443294 master-1 kubenswrapper[4771]: I1011 10:26:14.443111 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" event={"ID":"3273b5dc02e0d8cacbf64fe78c713d50","Type":"ContainerStarted","Data":"d3beeb8f52049cfbfd643d5240049be7138d9ff83afdc0d99012e206f07dfb78"} Oct 11 10:26:15.267616 master-1 kubenswrapper[4771]: I1011 10:26:15.267547 4771 csi_plugin.go:884] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "master-1" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 11 10:26:15.291127 master-1 kubenswrapper[4771]: E1011 10:26:15.291035 4771 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"master-1\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Oct 11 10:26:15.379937 master-1 kubenswrapper[4771]: W1011 10:26:15.379876 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User "system:anonymous" cannot list resource "runtimeclasses" in API group "node.k8s.io" at the cluster scope Oct 11 10:26:15.380096 master-1 kubenswrapper[4771]: E1011 10:26:15.379941 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:15.419418 master-1 kubenswrapper[4771]: E1011 10:26:15.419199 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-master-1.186d68e8179ab840 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-master-1,UID:3273b5dc02e0d8cacbf64fe78c713d50,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f22b65e5c744a32d3955dd7c36d809e3114a8aa501b44c00330dfda886c21169\" in 1.687s (1.687s including waiting). Image size: 458126368 bytes.,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:15.409449024 +0000 UTC m=+7.383675495,LastTimestamp:2025-10-11 10:26:15.409449024 +0000 UTC m=+7.383675495,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:15.587782 master-1 kubenswrapper[4771]: I1011 10:26:15.587689 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:15.589489 master-1 kubenswrapper[4771]: I1011 10:26:15.589444 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:15.589489 master-1 kubenswrapper[4771]: I1011 10:26:15.589485 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:15.589489 master-1 kubenswrapper[4771]: I1011 10:26:15.589493 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:15.589675 master-1 kubenswrapper[4771]: I1011 10:26:15.589526 4771 kubelet_node_status.go:76] "Attempting to register node" node="master-1" Oct 11 10:26:15.600624 master-1 kubenswrapper[4771]: E1011 10:26:15.600566 4771 kubelet_node_status.go:99] "Unable to register node with API server" err="nodes is forbidden: User \"system:anonymous\" cannot create resource \"nodes\" in API group \"\" at the cluster scope" node="master-1" Oct 11 10:26:15.600813 master-1 kubenswrapper[4771]: E1011 10:26:15.600691 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e726697\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e726697 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node master-1 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303021719 +0000 UTC m=+4.277248170,LastTimestamp:2025-10-11 10:26:15.589474669 +0000 UTC m=+7.563701110,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:15.611136 master-1 kubenswrapper[4771]: E1011 10:26:15.611047 4771 event.go:359] "Server rejected event (will not retry!)" err="events \"master-1.186d68e75e72c42f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{master-1.186d68e75e72c42f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:master-1,UID:master-1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node master-1 status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:12.303045679 +0000 UTC m=+4.277272130,LastTimestamp:2025-10-11 10:26:15.58949071 +0000 UTC m=+7.563717151,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:15.671151 master-1 kubenswrapper[4771]: E1011 10:26:15.670588 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-master-1.186d68e82685674f openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-master-1,UID:3273b5dc02e0d8cacbf64fe78c713d50,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:15.659710287 +0000 UTC m=+7.633936768,LastTimestamp:2025-10-11 10:26:15.659710287 +0000 UTC m=+7.633936768,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:15.685793 master-1 kubenswrapper[4771]: E1011 10:26:15.685595 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-master-1.186d68e82779c569 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-master-1,UID:3273b5dc02e0d8cacbf64fe78c713d50,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:15.675725161 +0000 UTC m=+7.649951612,LastTimestamp:2025-10-11 10:26:15.675725161 +0000 UTC m=+7.649951612,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:15.741838 master-1 kubenswrapper[4771]: W1011 10:26:15.741775 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope Oct 11 10:26:15.741838 master-1 kubenswrapper[4771]: E1011 10:26:15.741854 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:15.780891 master-1 kubenswrapper[4771]: W1011 10:26:15.780838 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope Oct 11 10:26:15.780891 master-1 kubenswrapper[4771]: E1011 10:26:15.780882 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:16.268109 master-1 kubenswrapper[4771]: I1011 10:26:16.268022 4771 csi_plugin.go:884] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "master-1" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 11 10:26:16.450977 master-1 kubenswrapper[4771]: I1011 10:26:16.450883 4771 generic.go:334] "Generic (PLEG): container finished" podID="3273b5dc02e0d8cacbf64fe78c713d50" containerID="1bfd32ecba3d08834d22892fd6303dea33b14629ed98fdcf0e18d648b3722608" exitCode=0 Oct 11 10:26:16.451262 master-1 kubenswrapper[4771]: I1011 10:26:16.450962 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" event={"ID":"3273b5dc02e0d8cacbf64fe78c713d50","Type":"ContainerDied","Data":"1bfd32ecba3d08834d22892fd6303dea33b14629ed98fdcf0e18d648b3722608"} Oct 11 10:26:16.451262 master-1 kubenswrapper[4771]: I1011 10:26:16.451032 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:16.452482 master-1 kubenswrapper[4771]: I1011 10:26:16.452417 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:16.452552 master-1 kubenswrapper[4771]: I1011 10:26:16.452497 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:16.452552 master-1 kubenswrapper[4771]: I1011 10:26:16.452506 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:16.479857 master-1 kubenswrapper[4771]: E1011 10:26:16.479666 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-master-1.186d68e856b5236d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-master-1,UID:3273b5dc02e0d8cacbf64fe78c713d50,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{kube-rbac-proxy-crio},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f22b65e5c744a32d3955dd7c36d809e3114a8aa501b44c00330dfda886c21169\" already present on machine,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:16.468145005 +0000 UTC m=+8.442371486,LastTimestamp:2025-10-11 10:26:16.468145005 +0000 UTC m=+8.442371486,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:16.539691 master-1 kubenswrapper[4771]: W1011 10:26:16.539542 4771 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Node: nodes "master-1" is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope Oct 11 10:26:16.539691 master-1 kubenswrapper[4771]: E1011 10:26:16.539614 4771 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Node: failed to list *v1.Node: nodes \"master-1\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" Oct 11 10:26:16.754830 master-1 kubenswrapper[4771]: E1011 10:26:16.754633 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-master-1.186d68e8671c480a openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-master-1,UID:3273b5dc02e0d8cacbf64fe78c713d50,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{kube-rbac-proxy-crio},},Reason:Created,Message:Created container: kube-rbac-proxy-crio,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:16.743340042 +0000 UTC m=+8.717566523,LastTimestamp:2025-10-11 10:26:16.743340042 +0000 UTC m=+8.717566523,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:16.770013 master-1 kubenswrapper[4771]: E1011 10:26:16.769833 4771 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-master-1.186d68e8682ca4e5 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-master-1,UID:3273b5dc02e0d8cacbf64fe78c713d50,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{kube-rbac-proxy-crio},},Reason:Started,Message:Started container kube-rbac-proxy-crio,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:26:16.761189605 +0000 UTC m=+8.735416086,LastTimestamp:2025-10-11 10:26:16.761189605 +0000 UTC m=+8.735416086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:26:16.892175 master-1 kubenswrapper[4771]: I1011 10:26:16.892074 4771 csr.go:261] certificate signing request csr-gzbc5 is approved, waiting to be issued Oct 11 10:26:16.901452 master-1 kubenswrapper[4771]: I1011 10:26:16.901335 4771 csr.go:257] certificate signing request csr-gzbc5 is issued Oct 11 10:26:17.121158 master-1 kubenswrapper[4771]: I1011 10:26:17.120888 4771 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 11 10:26:17.275634 master-1 kubenswrapper[4771]: I1011 10:26:17.275514 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:17.294139 master-1 kubenswrapper[4771]: I1011 10:26:17.294041 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:17.355253 master-1 kubenswrapper[4771]: I1011 10:26:17.355125 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:17.455426 master-1 kubenswrapper[4771]: I1011 10:26:17.455183 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-master-1_3273b5dc02e0d8cacbf64fe78c713d50/kube-rbac-proxy-crio/0.log" Oct 11 10:26:17.456121 master-1 kubenswrapper[4771]: I1011 10:26:17.456007 4771 generic.go:334] "Generic (PLEG): container finished" podID="3273b5dc02e0d8cacbf64fe78c713d50" containerID="28e895da0baa6877f335970836c88af88fc281c9171fd5ced86e8b5c1c9f3b5c" exitCode=1 Oct 11 10:26:17.456121 master-1 kubenswrapper[4771]: I1011 10:26:17.456112 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" event={"ID":"3273b5dc02e0d8cacbf64fe78c713d50","Type":"ContainerDied","Data":"28e895da0baa6877f335970836c88af88fc281c9171fd5ced86e8b5c1c9f3b5c"} Oct 11 10:26:17.456304 master-1 kubenswrapper[4771]: I1011 10:26:17.456166 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:17.457362 master-1 kubenswrapper[4771]: I1011 10:26:17.457298 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:17.457362 master-1 kubenswrapper[4771]: I1011 10:26:17.457342 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:17.457362 master-1 kubenswrapper[4771]: I1011 10:26:17.457364 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:17.471947 master-1 kubenswrapper[4771]: I1011 10:26:17.471867 4771 scope.go:117] "RemoveContainer" containerID="28e895da0baa6877f335970836c88af88fc281c9171fd5ced86e8b5c1c9f3b5c" Oct 11 10:26:17.621590 master-1 kubenswrapper[4771]: I1011 10:26:17.621494 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:17.621590 master-1 kubenswrapper[4771]: E1011 10:26:17.621564 4771 csi_plugin.go:305] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "master-1" not found Oct 11 10:26:17.644620 master-1 kubenswrapper[4771]: I1011 10:26:17.644549 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:17.663167 master-1 kubenswrapper[4771]: I1011 10:26:17.663067 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:17.723246 master-1 kubenswrapper[4771]: I1011 10:26:17.723180 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:17.904008 master-1 kubenswrapper[4771]: I1011 10:26:17.903873 4771 certificate_manager.go:356] kubernetes.io/kube-apiserver-client-kubelet: Certificate expiration is 2025-10-12 10:21:12 +0000 UTC, rotation deadline is 2025-10-12 04:52:52.049163029 +0000 UTC Oct 11 10:26:17.904008 master-1 kubenswrapper[4771]: I1011 10:26:17.903945 4771 certificate_manager.go:356] kubernetes.io/kube-apiserver-client-kubelet: Waiting 18h26m34.145220968s for next certificate rotation Oct 11 10:26:17.996177 master-1 kubenswrapper[4771]: I1011 10:26:17.995991 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:17.996177 master-1 kubenswrapper[4771]: E1011 10:26:17.996042 4771 csi_plugin.go:305] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "master-1" not found Oct 11 10:26:18.100463 master-1 kubenswrapper[4771]: I1011 10:26:18.100342 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:18.117890 master-1 kubenswrapper[4771]: I1011 10:26:18.117832 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:18.177844 master-1 kubenswrapper[4771]: I1011 10:26:18.177787 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:18.456455 master-1 kubenswrapper[4771]: I1011 10:26:18.456400 4771 nodeinfomanager.go:401] Failed to publish CSINode: nodes "master-1" not found Oct 11 10:26:18.456455 master-1 kubenswrapper[4771]: E1011 10:26:18.456454 4771 csi_plugin.go:305] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "master-1" not found Oct 11 10:26:18.460026 master-1 kubenswrapper[4771]: I1011 10:26:18.459941 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-master-1_3273b5dc02e0d8cacbf64fe78c713d50/kube-rbac-proxy-crio/1.log" Oct 11 10:26:18.460707 master-1 kubenswrapper[4771]: I1011 10:26:18.460661 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-master-1_3273b5dc02e0d8cacbf64fe78c713d50/kube-rbac-proxy-crio/0.log" Oct 11 10:26:18.461286 master-1 kubenswrapper[4771]: I1011 10:26:18.461226 4771 generic.go:334] "Generic (PLEG): container finished" podID="3273b5dc02e0d8cacbf64fe78c713d50" containerID="a8e5e0132e090b8e8d3d5ec1f39e48dd2fe3f756d595c30bb279ad118a40f824" exitCode=1 Oct 11 10:26:18.461357 master-1 kubenswrapper[4771]: I1011 10:26:18.461291 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" event={"ID":"3273b5dc02e0d8cacbf64fe78c713d50","Type":"ContainerDied","Data":"a8e5e0132e090b8e8d3d5ec1f39e48dd2fe3f756d595c30bb279ad118a40f824"} Oct 11 10:26:18.461472 master-1 kubenswrapper[4771]: I1011 10:26:18.461365 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:18.461547 master-1 kubenswrapper[4771]: I1011 10:26:18.461399 4771 scope.go:117] "RemoveContainer" containerID="28e895da0baa6877f335970836c88af88fc281c9171fd5ced86e8b5c1c9f3b5c" Oct 11 10:26:18.463000 master-1 kubenswrapper[4771]: I1011 10:26:18.462941 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:18.463095 master-1 kubenswrapper[4771]: I1011 10:26:18.463006 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:18.463095 master-1 kubenswrapper[4771]: I1011 10:26:18.463025 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:18.481955 master-1 kubenswrapper[4771]: I1011 10:26:18.481900 4771 scope.go:117] "RemoveContainer" containerID="a8e5e0132e090b8e8d3d5ec1f39e48dd2fe3f756d595c30bb279ad118a40f824" Oct 11 10:26:18.482214 master-1 kubenswrapper[4771]: E1011 10:26:18.482163 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy-crio\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-rbac-proxy-crio pod=kube-rbac-proxy-crio-master-1_openshift-machine-config-operator(3273b5dc02e0d8cacbf64fe78c713d50)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" podUID="3273b5dc02e0d8cacbf64fe78c713d50" Oct 11 10:26:18.497949 master-1 kubenswrapper[4771]: E1011 10:26:18.497896 4771 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"master-1\" not found" node="master-1" Oct 11 10:26:18.801240 master-1 kubenswrapper[4771]: I1011 10:26:18.801006 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:18.803415 master-1 kubenswrapper[4771]: I1011 10:26:18.803342 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:18.803484 master-1 kubenswrapper[4771]: I1011 10:26:18.803430 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:18.803484 master-1 kubenswrapper[4771]: I1011 10:26:18.803449 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:18.803540 master-1 kubenswrapper[4771]: I1011 10:26:18.803495 4771 kubelet_node_status.go:76] "Attempting to register node" node="master-1" Oct 11 10:26:18.813766 master-1 kubenswrapper[4771]: I1011 10:26:18.813693 4771 kubelet_node_status.go:79] "Successfully registered node" node="master-1" Oct 11 10:26:18.813819 master-1 kubenswrapper[4771]: E1011 10:26:18.813777 4771 kubelet_node_status.go:585] "Error updating node status, will retry" err="error getting node \"master-1\": node \"master-1\" not found" Oct 11 10:26:18.837812 master-1 kubenswrapper[4771]: E1011 10:26:18.837716 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:18.938496 master-1 kubenswrapper[4771]: E1011 10:26:18.938274 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.039014 master-1 kubenswrapper[4771]: E1011 10:26:19.038832 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.140204 master-1 kubenswrapper[4771]: E1011 10:26:19.140096 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.241212 master-1 kubenswrapper[4771]: E1011 10:26:19.241103 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.254842 master-1 kubenswrapper[4771]: I1011 10:26:19.254785 4771 certificate_manager.go:356] kubernetes.io/kubelet-serving: Rotating certificates Oct 11 10:26:19.271392 master-1 kubenswrapper[4771]: I1011 10:26:19.271318 4771 reflector.go:368] Caches populated for *v1.CertificateSigningRequest from k8s.io/client-go/tools/watch/informerwatcher.go:146 Oct 11 10:26:19.341728 master-1 kubenswrapper[4771]: E1011 10:26:19.341560 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.441979 master-1 kubenswrapper[4771]: E1011 10:26:19.441758 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.465748 master-1 kubenswrapper[4771]: I1011 10:26:19.465663 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-master-1_3273b5dc02e0d8cacbf64fe78c713d50/kube-rbac-proxy-crio/1.log" Oct 11 10:26:19.467357 master-1 kubenswrapper[4771]: I1011 10:26:19.467293 4771 kubelet_node_status.go:401] "Setting node annotation to enable volume controller attach/detach" Oct 11 10:26:19.468567 master-1 kubenswrapper[4771]: I1011 10:26:19.468499 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientMemory" Oct 11 10:26:19.468567 master-1 kubenswrapper[4771]: I1011 10:26:19.468547 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasNoDiskPressure" Oct 11 10:26:19.468567 master-1 kubenswrapper[4771]: I1011 10:26:19.468563 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeHasSufficientPID" Oct 11 10:26:19.469002 master-1 kubenswrapper[4771]: I1011 10:26:19.468937 4771 scope.go:117] "RemoveContainer" containerID="a8e5e0132e090b8e8d3d5ec1f39e48dd2fe3f756d595c30bb279ad118a40f824" Oct 11 10:26:19.469189 master-1 kubenswrapper[4771]: E1011 10:26:19.469152 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy-crio\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-rbac-proxy-crio pod=kube-rbac-proxy-crio-master-1_openshift-machine-config-operator(3273b5dc02e0d8cacbf64fe78c713d50)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" podUID="3273b5dc02e0d8cacbf64fe78c713d50" Oct 11 10:26:19.542142 master-1 kubenswrapper[4771]: E1011 10:26:19.541971 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.643399 master-1 kubenswrapper[4771]: E1011 10:26:19.643260 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.744730 master-1 kubenswrapper[4771]: E1011 10:26:19.744430 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.844923 master-1 kubenswrapper[4771]: E1011 10:26:19.844804 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:19.945968 master-1 kubenswrapper[4771]: E1011 10:26:19.945815 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.046606 master-1 kubenswrapper[4771]: E1011 10:26:20.046308 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.147722 master-1 kubenswrapper[4771]: E1011 10:26:20.147588 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.248669 master-1 kubenswrapper[4771]: E1011 10:26:20.248548 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.349215 master-1 kubenswrapper[4771]: E1011 10:26:20.349048 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.450176 master-1 kubenswrapper[4771]: E1011 10:26:20.450090 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.551499 master-1 kubenswrapper[4771]: E1011 10:26:20.551305 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.651971 master-1 kubenswrapper[4771]: E1011 10:26:20.651753 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.753020 master-1 kubenswrapper[4771]: E1011 10:26:20.752871 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.853738 master-1 kubenswrapper[4771]: E1011 10:26:20.853656 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:20.870411 master-1 kubenswrapper[4771]: I1011 10:26:20.870321 4771 reflector.go:368] Caches populated for *v1.Service from k8s.io/client-go/informers/factory.go:160 Oct 11 10:26:20.954140 master-1 kubenswrapper[4771]: E1011 10:26:20.953844 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.054686 master-1 kubenswrapper[4771]: E1011 10:26:21.054567 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.154961 master-1 kubenswrapper[4771]: E1011 10:26:21.154838 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.256178 master-1 kubenswrapper[4771]: E1011 10:26:21.255944 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.357315 master-1 kubenswrapper[4771]: E1011 10:26:21.357168 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.457890 master-1 kubenswrapper[4771]: E1011 10:26:21.457738 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.535596 master-1 kubenswrapper[4771]: I1011 10:26:21.535403 4771 reflector.go:368] Caches populated for *v1.RuntimeClass from k8s.io/client-go/informers/factory.go:160 Oct 11 10:26:21.558388 master-1 kubenswrapper[4771]: E1011 10:26:21.558295 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.659317 master-1 kubenswrapper[4771]: E1011 10:26:21.659199 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.759537 master-1 kubenswrapper[4771]: E1011 10:26:21.759405 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.859815 master-1 kubenswrapper[4771]: E1011 10:26:21.859678 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.950074 master-1 kubenswrapper[4771]: I1011 10:26:21.949961 4771 reflector.go:368] Caches populated for *v1.CSIDriver from k8s.io/client-go/informers/factory.go:160 Oct 11 10:26:21.960951 master-1 kubenswrapper[4771]: E1011 10:26:21.960872 4771 kubelet_node_status.go:503] "Error getting the current node from lister" err="node \"master-1\" not found" Oct 11 10:26:21.997817 master-1 kubenswrapper[4771]: I1011 10:26:21.997697 4771 reflector.go:368] Caches populated for *v1.Node from k8s.io/client-go/informers/factory.go:160 Oct 11 10:26:22.236943 master-1 kubenswrapper[4771]: I1011 10:26:22.236779 4771 apiserver.go:52] "Watching apiserver" Oct 11 10:26:22.242485 master-1 kubenswrapper[4771]: I1011 10:26:22.242442 4771 reflector.go:368] Caches populated for *v1.Pod from pkg/kubelet/config/apiserver.go:66 Oct 11 10:26:22.242635 master-1 kubenswrapper[4771]: I1011 10:26:22.242600 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx"] Oct 11 10:26:22.242851 master-1 kubenswrapper[4771]: I1011 10:26:22.242826 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.246391 master-1 kubenswrapper[4771]: I1011 10:26:22.246337 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-cluster-version"/"cluster-version-operator-serving-cert" Oct 11 10:26:22.246640 master-1 kubenswrapper[4771]: I1011 10:26:22.246575 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cluster-version"/"openshift-service-ca.crt" Oct 11 10:26:22.246820 master-1 kubenswrapper[4771]: I1011 10:26:22.246772 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cluster-version"/"kube-root-ca.crt" Oct 11 10:26:22.250513 master-1 kubenswrapper[4771]: I1011 10:26:22.250477 4771 desired_state_of_world_populator.go:155] "Finished populating initial desired state of world" Oct 11 10:26:22.318018 master-1 kubenswrapper[4771]: I1011 10:26:22.317895 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-ssl-certs\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-ssl-certs\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.318018 master-1 kubenswrapper[4771]: I1011 10:26:22.318003 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-cvo-updatepayloads\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-cvo-updatepayloads\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.318579 master-1 kubenswrapper[4771]: I1011 10:26:22.318154 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.318579 master-1 kubenswrapper[4771]: I1011 10:26:22.318248 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68d07184-647e-4aaa-a3e6-85e99ae0abd2-service-ca\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.318579 master-1 kubenswrapper[4771]: I1011 10:26:22.318283 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/68d07184-647e-4aaa-a3e6-85e99ae0abd2-kube-api-access\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.419542 master-1 kubenswrapper[4771]: I1011 10:26:22.419439 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/68d07184-647e-4aaa-a3e6-85e99ae0abd2-kube-api-access\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.419542 master-1 kubenswrapper[4771]: I1011 10:26:22.419531 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-ssl-certs\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-ssl-certs\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.419948 master-1 kubenswrapper[4771]: I1011 10:26:22.419565 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-cvo-updatepayloads\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-cvo-updatepayloads\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.419948 master-1 kubenswrapper[4771]: I1011 10:26:22.419598 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.419948 master-1 kubenswrapper[4771]: I1011 10:26:22.419630 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68d07184-647e-4aaa-a3e6-85e99ae0abd2-service-ca\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.419948 master-1 kubenswrapper[4771]: I1011 10:26:22.419768 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-ssl-certs\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-ssl-certs\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.419948 master-1 kubenswrapper[4771]: I1011 10:26:22.419802 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-cvo-updatepayloads\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-cvo-updatepayloads\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.420258 master-1 kubenswrapper[4771]: E1011 10:26:22.420050 4771 secret.go:189] Couldn't get secret openshift-cluster-version/cluster-version-operator-serving-cert: secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:22.420258 master-1 kubenswrapper[4771]: E1011 10:26:22.420195 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert podName:68d07184-647e-4aaa-a3e6-85e99ae0abd2 nodeName:}" failed. No retries permitted until 2025-10-11 10:26:22.920157856 +0000 UTC m=+14.894384327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert") pod "cluster-version-operator-55ccd5d5cf-mqqvx" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2") : secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:22.421192 master-1 kubenswrapper[4771]: I1011 10:26:22.421138 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68d07184-647e-4aaa-a3e6-85e99ae0abd2-service-ca\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.446614 master-1 kubenswrapper[4771]: I1011 10:26:22.446497 4771 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 11 10:26:22.453873 master-1 kubenswrapper[4771]: I1011 10:26:22.453829 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/68d07184-647e-4aaa-a3e6-85e99ae0abd2-kube-api-access\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.935729 master-1 kubenswrapper[4771]: I1011 10:26:22.935614 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:22.936939 master-1 kubenswrapper[4771]: E1011 10:26:22.935847 4771 secret.go:189] Couldn't get secret openshift-cluster-version/cluster-version-operator-serving-cert: secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:22.936939 master-1 kubenswrapper[4771]: E1011 10:26:22.936007 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert podName:68d07184-647e-4aaa-a3e6-85e99ae0abd2 nodeName:}" failed. No retries permitted until 2025-10-11 10:26:23.935975419 +0000 UTC m=+15.910201890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert") pod "cluster-version-operator-55ccd5d5cf-mqqvx" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2") : secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:23.941504 master-1 kubenswrapper[4771]: I1011 10:26:23.941396 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:23.941504 master-1 kubenswrapper[4771]: E1011 10:26:23.941464 4771 secret.go:189] Couldn't get secret openshift-cluster-version/cluster-version-operator-serving-cert: secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:23.942220 master-1 kubenswrapper[4771]: E1011 10:26:23.941555 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert podName:68d07184-647e-4aaa-a3e6-85e99ae0abd2 nodeName:}" failed. No retries permitted until 2025-10-11 10:26:25.941530878 +0000 UTC m=+17.915757349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert") pod "cluster-version-operator-55ccd5d5cf-mqqvx" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2") : secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:25.955046 master-1 kubenswrapper[4771]: I1011 10:26:25.954826 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:25.955867 master-1 kubenswrapper[4771]: E1011 10:26:25.955073 4771 secret.go:189] Couldn't get secret openshift-cluster-version/cluster-version-operator-serving-cert: secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:25.955867 master-1 kubenswrapper[4771]: E1011 10:26:25.955190 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert podName:68d07184-647e-4aaa-a3e6-85e99ae0abd2 nodeName:}" failed. No retries permitted until 2025-10-11 10:26:29.955169872 +0000 UTC m=+21.929396313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert") pod "cluster-version-operator-55ccd5d5cf-mqqvx" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2") : secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:29.989610 master-1 kubenswrapper[4771]: I1011 10:26:29.989477 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:29.989610 master-1 kubenswrapper[4771]: E1011 10:26:29.989622 4771 secret.go:189] Couldn't get secret openshift-cluster-version/cluster-version-operator-serving-cert: secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:29.990610 master-1 kubenswrapper[4771]: E1011 10:26:29.989675 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert podName:68d07184-647e-4aaa-a3e6-85e99ae0abd2 nodeName:}" failed. No retries permitted until 2025-10-11 10:26:37.989657863 +0000 UTC m=+29.963884304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert") pod "cluster-version-operator-55ccd5d5cf-mqqvx" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2") : secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:31.447666 master-1 kubenswrapper[4771]: I1011 10:26:31.447561 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-master-1"] Oct 11 10:26:31.448267 master-1 kubenswrapper[4771]: I1011 10:26:31.447792 4771 scope.go:117] "RemoveContainer" containerID="a8e5e0132e090b8e8d3d5ec1f39e48dd2fe3f756d595c30bb279ad118a40f824" Oct 11 10:26:32.497911 master-1 kubenswrapper[4771]: I1011 10:26:32.497832 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-master-1_3273b5dc02e0d8cacbf64fe78c713d50/kube-rbac-proxy-crio/2.log" Oct 11 10:26:32.498855 master-1 kubenswrapper[4771]: I1011 10:26:32.498777 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-master-1_3273b5dc02e0d8cacbf64fe78c713d50/kube-rbac-proxy-crio/1.log" Oct 11 10:26:32.499291 master-1 kubenswrapper[4771]: I1011 10:26:32.499237 4771 generic.go:334] "Generic (PLEG): container finished" podID="3273b5dc02e0d8cacbf64fe78c713d50" containerID="a32ecb2841115f637b624a3f8eaa25803703a8d4195c571015a3a44c7767232c" exitCode=1 Oct 11 10:26:32.499291 master-1 kubenswrapper[4771]: I1011 10:26:32.499285 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" event={"ID":"3273b5dc02e0d8cacbf64fe78c713d50","Type":"ContainerDied","Data":"a32ecb2841115f637b624a3f8eaa25803703a8d4195c571015a3a44c7767232c"} Oct 11 10:26:32.499456 master-1 kubenswrapper[4771]: I1011 10:26:32.499333 4771 scope.go:117] "RemoveContainer" containerID="a8e5e0132e090b8e8d3d5ec1f39e48dd2fe3f756d595c30bb279ad118a40f824" Oct 11 10:26:32.510519 master-1 kubenswrapper[4771]: I1011 10:26:32.510476 4771 scope.go:117] "RemoveContainer" containerID="a32ecb2841115f637b624a3f8eaa25803703a8d4195c571015a3a44c7767232c" Oct 11 10:26:32.510974 master-1 kubenswrapper[4771]: E1011 10:26:32.510938 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy-crio\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-rbac-proxy-crio pod=kube-rbac-proxy-crio-master-1_openshift-machine-config-operator(3273b5dc02e0d8cacbf64fe78c713d50)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" podUID="3273b5dc02e0d8cacbf64fe78c713d50" Oct 11 10:26:33.504615 master-1 kubenswrapper[4771]: I1011 10:26:33.504498 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-master-1_3273b5dc02e0d8cacbf64fe78c713d50/kube-rbac-proxy-crio/2.log" Oct 11 10:26:33.505939 master-1 kubenswrapper[4771]: I1011 10:26:33.505877 4771 scope.go:117] "RemoveContainer" containerID="a32ecb2841115f637b624a3f8eaa25803703a8d4195c571015a3a44c7767232c" Oct 11 10:26:33.506201 master-1 kubenswrapper[4771]: E1011 10:26:33.506150 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy-crio\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-rbac-proxy-crio pod=kube-rbac-proxy-crio-master-1_openshift-machine-config-operator(3273b5dc02e0d8cacbf64fe78c713d50)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" podUID="3273b5dc02e0d8cacbf64fe78c713d50" Oct 11 10:26:37.304004 master-1 kubenswrapper[4771]: I1011 10:26:37.303867 4771 csr.go:261] certificate signing request csr-sdd7k is approved, waiting to be issued Oct 11 10:26:37.313223 master-1 kubenswrapper[4771]: I1011 10:26:37.313176 4771 csr.go:257] certificate signing request csr-sdd7k is issued Oct 11 10:26:38.042848 master-1 kubenswrapper[4771]: I1011 10:26:38.042740 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert\") pod \"cluster-version-operator-55ccd5d5cf-mqqvx\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:38.043238 master-1 kubenswrapper[4771]: E1011 10:26:38.042935 4771 secret.go:189] Couldn't get secret openshift-cluster-version/cluster-version-operator-serving-cert: secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:38.043238 master-1 kubenswrapper[4771]: E1011 10:26:38.043051 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert podName:68d07184-647e-4aaa-a3e6-85e99ae0abd2 nodeName:}" failed. No retries permitted until 2025-10-11 10:26:54.043016511 +0000 UTC m=+46.017242992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert") pod "cluster-version-operator-55ccd5d5cf-mqqvx" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2") : secret "cluster-version-operator-serving-cert" not found Oct 11 10:26:38.315078 master-1 kubenswrapper[4771]: I1011 10:26:38.314972 4771 certificate_manager.go:356] kubernetes.io/kubelet-serving: Certificate expiration is 2025-10-12 10:21:12 +0000 UTC, rotation deadline is 2025-10-12 06:07:40.78853197 +0000 UTC Oct 11 10:26:38.315078 master-1 kubenswrapper[4771]: I1011 10:26:38.315018 4771 certificate_manager.go:356] kubernetes.io/kubelet-serving: Waiting 19h41m2.473517472s for next certificate rotation Oct 11 10:26:39.315969 master-1 kubenswrapper[4771]: I1011 10:26:39.315805 4771 certificate_manager.go:356] kubernetes.io/kubelet-serving: Certificate expiration is 2025-10-12 10:21:12 +0000 UTC, rotation deadline is 2025-10-12 06:51:24.981270668 +0000 UTC Oct 11 10:26:39.315969 master-1 kubenswrapper[4771]: I1011 10:26:39.315875 4771 certificate_manager.go:356] kubernetes.io/kubelet-serving: Waiting 20h24m45.66539847s for next certificate rotation Oct 11 10:26:44.967446 master-1 kubenswrapper[4771]: I1011 10:26:44.967382 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx"] Oct 11 10:26:44.968577 master-1 kubenswrapper[4771]: E1011 10:26:44.967538 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[serving-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" podUID="68d07184-647e-4aaa-a3e6-85e99ae0abd2" Oct 11 10:26:45.527550 master-1 kubenswrapper[4771]: I1011 10:26:45.527490 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:45.548239 master-1 kubenswrapper[4771]: I1011 10:26:45.548160 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:45.692424 master-1 kubenswrapper[4771]: I1011 10:26:45.692290 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etc-ssl-certs\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-ssl-certs\") pod \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " Oct 11 10:26:45.692424 master-1 kubenswrapper[4771]: I1011 10:26:45.692387 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etc-cvo-updatepayloads\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-cvo-updatepayloads\") pod \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " Oct 11 10:26:45.692424 master-1 kubenswrapper[4771]: I1011 10:26:45.692432 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68d07184-647e-4aaa-a3e6-85e99ae0abd2-service-ca\") pod \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " Oct 11 10:26:45.692888 master-1 kubenswrapper[4771]: I1011 10:26:45.692470 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/68d07184-647e-4aaa-a3e6-85e99ae0abd2-kube-api-access\") pod \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\" (UID: \"68d07184-647e-4aaa-a3e6-85e99ae0abd2\") " Oct 11 10:26:45.692888 master-1 kubenswrapper[4771]: I1011 10:26:45.692565 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-ssl-certs" (OuterVolumeSpecName: "etc-ssl-certs") pod "68d07184-647e-4aaa-a3e6-85e99ae0abd2" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2"). InnerVolumeSpecName "etc-ssl-certs". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:26:45.692888 master-1 kubenswrapper[4771]: I1011 10:26:45.692583 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-cvo-updatepayloads" (OuterVolumeSpecName: "etc-cvo-updatepayloads") pod "68d07184-647e-4aaa-a3e6-85e99ae0abd2" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2"). InnerVolumeSpecName "etc-cvo-updatepayloads". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:26:45.693344 master-1 kubenswrapper[4771]: I1011 10:26:45.693276 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d07184-647e-4aaa-a3e6-85e99ae0abd2-service-ca" (OuterVolumeSpecName: "service-ca") pod "68d07184-647e-4aaa-a3e6-85e99ae0abd2" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:26:45.698153 master-1 kubenswrapper[4771]: I1011 10:26:45.698087 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d07184-647e-4aaa-a3e6-85e99ae0abd2-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "68d07184-647e-4aaa-a3e6-85e99ae0abd2" (UID: "68d07184-647e-4aaa-a3e6-85e99ae0abd2"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:26:45.793757 master-1 kubenswrapper[4771]: I1011 10:26:45.793511 4771 reconciler_common.go:293] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68d07184-647e-4aaa-a3e6-85e99ae0abd2-service-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:26:45.793757 master-1 kubenswrapper[4771]: I1011 10:26:45.793585 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/68d07184-647e-4aaa-a3e6-85e99ae0abd2-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:26:45.793757 master-1 kubenswrapper[4771]: I1011 10:26:45.793604 4771 reconciler_common.go:293] "Volume detached for volume \"etc-ssl-certs\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-ssl-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:26:45.793757 master-1 kubenswrapper[4771]: I1011 10:26:45.793621 4771 reconciler_common.go:293] "Volume detached for volume \"etc-cvo-updatepayloads\" (UniqueName: \"kubernetes.io/host-path/68d07184-647e-4aaa-a3e6-85e99ae0abd2-etc-cvo-updatepayloads\") on node \"master-1\" DevicePath \"\"" Oct 11 10:26:46.529304 master-1 kubenswrapper[4771]: I1011 10:26:46.529200 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx" Oct 11 10:26:46.557433 master-1 kubenswrapper[4771]: I1011 10:26:46.557320 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx"] Oct 11 10:26:46.561558 master-1 kubenswrapper[4771]: I1011 10:26:46.561501 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-cluster-version/cluster-version-operator-55ccd5d5cf-mqqvx"] Oct 11 10:26:46.701208 master-1 kubenswrapper[4771]: I1011 10:26:46.701004 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d07184-647e-4aaa-a3e6-85e99ae0abd2-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:26:47.451189 master-1 kubenswrapper[4771]: I1011 10:26:47.451089 4771 scope.go:117] "RemoveContainer" containerID="a32ecb2841115f637b624a3f8eaa25803703a8d4195c571015a3a44c7767232c" Oct 11 10:26:47.451451 master-1 kubenswrapper[4771]: E1011 10:26:47.451403 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy-crio\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-rbac-proxy-crio pod=kube-rbac-proxy-crio-master-1_openshift-machine-config-operator(3273b5dc02e0d8cacbf64fe78c713d50)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" podUID="3273b5dc02e0d8cacbf64fe78c713d50" Oct 11 10:26:48.442026 master-1 kubenswrapper[4771]: I1011 10:26:48.441921 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d07184-647e-4aaa-a3e6-85e99ae0abd2" path="/var/lib/kubelet/pods/68d07184-647e-4aaa-a3e6-85e99ae0abd2/volumes" Oct 11 10:26:53.810165 master-1 kubenswrapper[4771]: I1011 10:26:53.810059 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf"] Oct 11 10:26:53.811340 master-1 kubenswrapper[4771]: I1011 10:26:53.810408 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:53.813811 master-1 kubenswrapper[4771]: I1011 10:26:53.813763 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cloud-controller-manager-operator"/"kube-root-ca.crt" Oct 11 10:26:53.814702 master-1 kubenswrapper[4771]: I1011 10:26:53.814638 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-cloud-controller-manager-operator"/"cloud-controller-manager-operator-tls" Oct 11 10:26:53.815135 master-1 kubenswrapper[4771]: I1011 10:26:53.815074 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cloud-controller-manager-operator"/"openshift-service-ca.crt" Oct 11 10:26:53.815334 master-1 kubenswrapper[4771]: I1011 10:26:53.815291 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cloud-controller-manager-operator"/"kube-rbac-proxy" Oct 11 10:26:53.815816 master-1 kubenswrapper[4771]: I1011 10:26:53.815764 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cloud-controller-manager-operator"/"cloud-controller-manager-images" Oct 11 10:26:53.947588 master-1 kubenswrapper[4771]: I1011 10:26:53.947467 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c85t\" (UniqueName: \"kubernetes.io/projected/4ba9953d-1f54-43be-a3ae-121030f1e07b-kube-api-access-4c85t\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:53.947588 master-1 kubenswrapper[4771]: I1011 10:26:53.947544 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"images\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-images\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:53.947588 master-1 kubenswrapper[4771]: I1011 10:26:53.947583 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ba9953d-1f54-43be-a3ae-121030f1e07b-host-etc-kube\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:53.947588 master-1 kubenswrapper[4771]: I1011 10:26:53.947616 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-auth-proxy-config\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:53.948097 master-1 kubenswrapper[4771]: I1011 10:26:53.947690 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cloud-controller-manager-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ba9953d-1f54-43be-a3ae-121030f1e07b-cloud-controller-manager-operator-tls\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.048900 master-1 kubenswrapper[4771]: I1011 10:26:54.048762 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ba9953d-1f54-43be-a3ae-121030f1e07b-host-etc-kube\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.048900 master-1 kubenswrapper[4771]: I1011 10:26:54.048860 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-auth-proxy-config\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.048900 master-1 kubenswrapper[4771]: I1011 10:26:54.048903 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cloud-controller-manager-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ba9953d-1f54-43be-a3ae-121030f1e07b-cloud-controller-manager-operator-tls\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.049257 master-1 kubenswrapper[4771]: I1011 10:26:54.048943 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-4c85t\" (UniqueName: \"kubernetes.io/projected/4ba9953d-1f54-43be-a3ae-121030f1e07b-kube-api-access-4c85t\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.049257 master-1 kubenswrapper[4771]: I1011 10:26:54.048979 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"images\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-images\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.049257 master-1 kubenswrapper[4771]: I1011 10:26:54.049046 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ba9953d-1f54-43be-a3ae-121030f1e07b-host-etc-kube\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.050278 master-1 kubenswrapper[4771]: I1011 10:26:54.050223 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"images\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-images\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.050385 master-1 kubenswrapper[4771]: I1011 10:26:54.050288 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-auth-proxy-config\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.054269 master-1 kubenswrapper[4771]: I1011 10:26:54.054195 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cloud-controller-manager-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ba9953d-1f54-43be-a3ae-121030f1e07b-cloud-controller-manager-operator-tls\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.079612 master-1 kubenswrapper[4771]: I1011 10:26:54.079491 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c85t\" (UniqueName: \"kubernetes.io/projected/4ba9953d-1f54-43be-a3ae-121030f1e07b-kube-api-access-4c85t\") pod \"cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.138033 master-1 kubenswrapper[4771]: I1011 10:26:54.137899 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:26:54.157007 master-1 kubenswrapper[4771]: W1011 10:26:54.156917 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba9953d_1f54_43be_a3ae_121030f1e07b.slice/crio-45041b48c16ec7268bc8f5e7bf6ad631a6a8600e9455c7fcc91bcbcacf5e65a9 WatchSource:0}: Error finding container 45041b48c16ec7268bc8f5e7bf6ad631a6a8600e9455c7fcc91bcbcacf5e65a9: Status 404 returned error can't find the container with id 45041b48c16ec7268bc8f5e7bf6ad631a6a8600e9455c7fcc91bcbcacf5e65a9 Oct 11 10:26:54.547864 master-1 kubenswrapper[4771]: I1011 10:26:54.547309 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" event={"ID":"4ba9953d-1f54-43be-a3ae-121030f1e07b","Type":"ContainerStarted","Data":"45041b48c16ec7268bc8f5e7bf6ad631a6a8600e9455c7fcc91bcbcacf5e65a9"} Oct 11 10:26:56.191780 master-1 kubenswrapper[4771]: I1011 10:26:56.191679 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-network-operator/network-operator-854f54f8c9-hw5fc"] Oct 11 10:26:56.192652 master-1 kubenswrapper[4771]: I1011 10:26:56.191895 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.194693 master-1 kubenswrapper[4771]: I1011 10:26:56.194598 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-operator"/"kube-root-ca.crt" Oct 11 10:26:56.194693 master-1 kubenswrapper[4771]: I1011 10:26:56.194617 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-operator"/"openshift-service-ca.crt" Oct 11 10:26:56.195119 master-1 kubenswrapper[4771]: I1011 10:26:56.194821 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-network-operator"/"metrics-tls" Oct 11 10:26:56.367930 master-1 kubenswrapper[4771]: I1011 10:26:56.367812 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/868ea5b9-b62a-4683-82c9-760de94ef155-host-etc-kube\") pod \"network-operator-854f54f8c9-hw5fc\" (UID: \"868ea5b9-b62a-4683-82c9-760de94ef155\") " pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.367930 master-1 kubenswrapper[4771]: I1011 10:26:56.367932 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/868ea5b9-b62a-4683-82c9-760de94ef155-metrics-tls\") pod \"network-operator-854f54f8c9-hw5fc\" (UID: \"868ea5b9-b62a-4683-82c9-760de94ef155\") " pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.368270 master-1 kubenswrapper[4771]: I1011 10:26:56.367985 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqrkd\" (UniqueName: \"kubernetes.io/projected/868ea5b9-b62a-4683-82c9-760de94ef155-kube-api-access-cqrkd\") pod \"network-operator-854f54f8c9-hw5fc\" (UID: \"868ea5b9-b62a-4683-82c9-760de94ef155\") " pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.468800 master-1 kubenswrapper[4771]: I1011 10:26:56.468665 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/868ea5b9-b62a-4683-82c9-760de94ef155-metrics-tls\") pod \"network-operator-854f54f8c9-hw5fc\" (UID: \"868ea5b9-b62a-4683-82c9-760de94ef155\") " pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.468800 master-1 kubenswrapper[4771]: I1011 10:26:56.468720 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-cqrkd\" (UniqueName: \"kubernetes.io/projected/868ea5b9-b62a-4683-82c9-760de94ef155-kube-api-access-cqrkd\") pod \"network-operator-854f54f8c9-hw5fc\" (UID: \"868ea5b9-b62a-4683-82c9-760de94ef155\") " pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.468800 master-1 kubenswrapper[4771]: I1011 10:26:56.468745 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/868ea5b9-b62a-4683-82c9-760de94ef155-host-etc-kube\") pod \"network-operator-854f54f8c9-hw5fc\" (UID: \"868ea5b9-b62a-4683-82c9-760de94ef155\") " pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.469092 master-1 kubenswrapper[4771]: I1011 10:26:56.468833 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/868ea5b9-b62a-4683-82c9-760de94ef155-host-etc-kube\") pod \"network-operator-854f54f8c9-hw5fc\" (UID: \"868ea5b9-b62a-4683-82c9-760de94ef155\") " pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.475149 master-1 kubenswrapper[4771]: I1011 10:26:56.475086 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/868ea5b9-b62a-4683-82c9-760de94ef155-metrics-tls\") pod \"network-operator-854f54f8c9-hw5fc\" (UID: \"868ea5b9-b62a-4683-82c9-760de94ef155\") " pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.498620 master-1 kubenswrapper[4771]: I1011 10:26:56.498551 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqrkd\" (UniqueName: \"kubernetes.io/projected/868ea5b9-b62a-4683-82c9-760de94ef155-kube-api-access-cqrkd\") pod \"network-operator-854f54f8c9-hw5fc\" (UID: \"868ea5b9-b62a-4683-82c9-760de94ef155\") " pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.502907 master-1 kubenswrapper[4771]: I1011 10:26:56.502874 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" Oct 11 10:26:56.635371 master-1 kubenswrapper[4771]: W1011 10:26:56.635300 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868ea5b9_b62a_4683_82c9_760de94ef155.slice/crio-8ee1d12f9e89c270226493db1b8178e230ee5b9a192f1ec9bdbb810e0d230446 WatchSource:0}: Error finding container 8ee1d12f9e89c270226493db1b8178e230ee5b9a192f1ec9bdbb810e0d230446: Status 404 returned error can't find the container with id 8ee1d12f9e89c270226493db1b8178e230ee5b9a192f1ec9bdbb810e0d230446 Oct 11 10:26:57.558336 master-1 kubenswrapper[4771]: I1011 10:26:57.558248 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_4ba9953d-1f54-43be-a3ae-121030f1e07b/kube-rbac-proxy/0.log" Oct 11 10:26:57.560317 master-1 kubenswrapper[4771]: I1011 10:26:57.560266 4771 generic.go:334] "Generic (PLEG): container finished" podID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerID="a1a2be8c23d4db9d020822dbf9ef90de797c8b8f057fc486ea46a8f7c185dd1e" exitCode=1 Oct 11 10:26:57.560525 master-1 kubenswrapper[4771]: I1011 10:26:57.560440 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" event={"ID":"4ba9953d-1f54-43be-a3ae-121030f1e07b","Type":"ContainerDied","Data":"a1a2be8c23d4db9d020822dbf9ef90de797c8b8f057fc486ea46a8f7c185dd1e"} Oct 11 10:26:57.560697 master-1 kubenswrapper[4771]: I1011 10:26:57.560530 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" event={"ID":"4ba9953d-1f54-43be-a3ae-121030f1e07b","Type":"ContainerStarted","Data":"ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62"} Oct 11 10:26:57.560697 master-1 kubenswrapper[4771]: I1011 10:26:57.560565 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" event={"ID":"4ba9953d-1f54-43be-a3ae-121030f1e07b","Type":"ContainerStarted","Data":"6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725"} Oct 11 10:26:57.561520 master-1 kubenswrapper[4771]: I1011 10:26:57.561466 4771 scope.go:117] "RemoveContainer" containerID="a1a2be8c23d4db9d020822dbf9ef90de797c8b8f057fc486ea46a8f7c185dd1e" Oct 11 10:26:57.562851 master-1 kubenswrapper[4771]: I1011 10:26:57.562795 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" event={"ID":"868ea5b9-b62a-4683-82c9-760de94ef155","Type":"ContainerStarted","Data":"8ee1d12f9e89c270226493db1b8178e230ee5b9a192f1ec9bdbb810e0d230446"} Oct 11 10:26:58.567666 master-1 kubenswrapper[4771]: I1011 10:26:58.567596 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_4ba9953d-1f54-43be-a3ae-121030f1e07b/kube-rbac-proxy/1.log" Oct 11 10:26:58.568487 master-1 kubenswrapper[4771]: I1011 10:26:58.568454 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_4ba9953d-1f54-43be-a3ae-121030f1e07b/kube-rbac-proxy/0.log" Oct 11 10:26:58.569978 master-1 kubenswrapper[4771]: I1011 10:26:58.569920 4771 generic.go:334] "Generic (PLEG): container finished" podID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerID="3fea68ffee0d3f0d9bbafc91305f308f57d148b9eb031b738b0ded06753f61e1" exitCode=1 Oct 11 10:26:58.570067 master-1 kubenswrapper[4771]: I1011 10:26:58.569978 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" event={"ID":"4ba9953d-1f54-43be-a3ae-121030f1e07b","Type":"ContainerDied","Data":"3fea68ffee0d3f0d9bbafc91305f308f57d148b9eb031b738b0ded06753f61e1"} Oct 11 10:26:58.570067 master-1 kubenswrapper[4771]: I1011 10:26:58.570020 4771 scope.go:117] "RemoveContainer" containerID="a1a2be8c23d4db9d020822dbf9ef90de797c8b8f057fc486ea46a8f7c185dd1e" Oct 11 10:26:58.570605 master-1 kubenswrapper[4771]: I1011 10:26:58.570557 4771 scope.go:117] "RemoveContainer" containerID="3fea68ffee0d3f0d9bbafc91305f308f57d148b9eb031b738b0ded06753f61e1" Oct 11 10:26:58.570824 master-1 kubenswrapper[4771]: E1011 10:26:58.570772 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_openshift-cloud-controller-manager-operator(4ba9953d-1f54-43be-a3ae-121030f1e07b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" Oct 11 10:26:59.436579 master-1 kubenswrapper[4771]: I1011 10:26:59.436510 4771 scope.go:117] "RemoveContainer" containerID="a32ecb2841115f637b624a3f8eaa25803703a8d4195c571015a3a44c7767232c" Oct 11 10:26:59.573480 master-1 kubenswrapper[4771]: I1011 10:26:59.573439 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_4ba9953d-1f54-43be-a3ae-121030f1e07b/kube-rbac-proxy/1.log" Oct 11 10:26:59.575734 master-1 kubenswrapper[4771]: I1011 10:26:59.575655 4771 scope.go:117] "RemoveContainer" containerID="3fea68ffee0d3f0d9bbafc91305f308f57d148b9eb031b738b0ded06753f61e1" Oct 11 10:26:59.576026 master-1 kubenswrapper[4771]: E1011 10:26:59.575984 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_openshift-cloud-controller-manager-operator(4ba9953d-1f54-43be-a3ae-121030f1e07b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" Oct 11 10:27:00.581095 master-1 kubenswrapper[4771]: I1011 10:27:00.581035 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-master-1_3273b5dc02e0d8cacbf64fe78c713d50/kube-rbac-proxy-crio/2.log" Oct 11 10:27:00.584893 master-1 kubenswrapper[4771]: I1011 10:27:00.584837 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" event={"ID":"3273b5dc02e0d8cacbf64fe78c713d50","Type":"ContainerStarted","Data":"c1c03abafd2dbbbe940e04d24ed5f7bce2acd566241e10c2566ec5ceea96e2b5"} Oct 11 10:27:01.592608 master-1 kubenswrapper[4771]: I1011 10:27:01.591553 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" event={"ID":"868ea5b9-b62a-4683-82c9-760de94ef155","Type":"ContainerStarted","Data":"081fde9dac0d8c6f0177a9a06139a4e92fb38ea47b03713cf7f04ea063469f84"} Oct 11 10:27:01.606420 master-1 kubenswrapper[4771]: I1011 10:27:01.606290 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-master-1" podStartSLOduration=30.606259031 podStartE2EDuration="30.606259031s" podCreationTimestamp="2025-10-11 10:26:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:27:00.603989263 +0000 UTC m=+52.578215734" watchObservedRunningTime="2025-10-11 10:27:01.606259031 +0000 UTC m=+53.580485512" Oct 11 10:27:01.606557 master-1 kubenswrapper[4771]: I1011 10:27:01.606479 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" podStartSLOduration=1.896275089 podStartE2EDuration="5.606471987s" podCreationTimestamp="2025-10-11 10:26:56 +0000 UTC" firstStartedPulling="2025-10-11 10:26:56.638287025 +0000 UTC m=+48.612513496" lastFinishedPulling="2025-10-11 10:27:00.348483953 +0000 UTC m=+52.322710394" observedRunningTime="2025-10-11 10:27:01.605929132 +0000 UTC m=+53.580155603" watchObservedRunningTime="2025-10-11 10:27:01.606471987 +0000 UTC m=+53.580698468" Oct 11 10:27:03.110790 master-1 kubenswrapper[4771]: I1011 10:27:03.110661 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-network-operator/mtu-prober-t9bz7"] Oct 11 10:27:03.111855 master-1 kubenswrapper[4771]: I1011 10:27:03.111029 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/mtu-prober-t9bz7" Oct 11 10:27:03.215600 master-1 kubenswrapper[4771]: I1011 10:27:03.215511 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw99n\" (UniqueName: \"kubernetes.io/projected/e7260b6d-3070-42b1-93cd-9ec29dfa50c3-kube-api-access-rw99n\") pod \"mtu-prober-t9bz7\" (UID: \"e7260b6d-3070-42b1-93cd-9ec29dfa50c3\") " pod="openshift-network-operator/mtu-prober-t9bz7" Oct 11 10:27:03.316563 master-1 kubenswrapper[4771]: I1011 10:27:03.316306 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-rw99n\" (UniqueName: \"kubernetes.io/projected/e7260b6d-3070-42b1-93cd-9ec29dfa50c3-kube-api-access-rw99n\") pod \"mtu-prober-t9bz7\" (UID: \"e7260b6d-3070-42b1-93cd-9ec29dfa50c3\") " pod="openshift-network-operator/mtu-prober-t9bz7" Oct 11 10:27:03.351238 master-1 kubenswrapper[4771]: I1011 10:27:03.351114 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw99n\" (UniqueName: \"kubernetes.io/projected/e7260b6d-3070-42b1-93cd-9ec29dfa50c3-kube-api-access-rw99n\") pod \"mtu-prober-t9bz7\" (UID: \"e7260b6d-3070-42b1-93cd-9ec29dfa50c3\") " pod="openshift-network-operator/mtu-prober-t9bz7" Oct 11 10:27:03.431319 master-1 kubenswrapper[4771]: I1011 10:27:03.431051 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/mtu-prober-t9bz7" Oct 11 10:27:03.450595 master-1 kubenswrapper[4771]: W1011 10:27:03.450490 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7260b6d_3070_42b1_93cd_9ec29dfa50c3.slice/crio-a6818c7d94ddd80836f979da47140d38cbf5ace31adebfef26025d803442a472 WatchSource:0}: Error finding container a6818c7d94ddd80836f979da47140d38cbf5ace31adebfef26025d803442a472: Status 404 returned error can't find the container with id a6818c7d94ddd80836f979da47140d38cbf5ace31adebfef26025d803442a472 Oct 11 10:27:03.599101 master-1 kubenswrapper[4771]: I1011 10:27:03.599009 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/mtu-prober-t9bz7" event={"ID":"e7260b6d-3070-42b1-93cd-9ec29dfa50c3","Type":"ContainerStarted","Data":"a6818c7d94ddd80836f979da47140d38cbf5ace31adebfef26025d803442a472"} Oct 11 10:27:04.603459 master-1 kubenswrapper[4771]: I1011 10:27:04.603336 4771 generic.go:334] "Generic (PLEG): container finished" podID="e7260b6d-3070-42b1-93cd-9ec29dfa50c3" containerID="a31d75d150e0d2dcf8878fd1b60bee95ea19d0157365ef6735168ff809442b4b" exitCode=0 Oct 11 10:27:04.603459 master-1 kubenswrapper[4771]: I1011 10:27:04.603412 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/mtu-prober-t9bz7" event={"ID":"e7260b6d-3070-42b1-93cd-9ec29dfa50c3","Type":"ContainerDied","Data":"a31d75d150e0d2dcf8878fd1b60bee95ea19d0157365ef6735168ff809442b4b"} Oct 11 10:27:05.631823 master-1 kubenswrapper[4771]: I1011 10:27:05.631710 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/mtu-prober-t9bz7" Oct 11 10:27:05.735228 master-1 kubenswrapper[4771]: I1011 10:27:05.735086 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw99n\" (UniqueName: \"kubernetes.io/projected/e7260b6d-3070-42b1-93cd-9ec29dfa50c3-kube-api-access-rw99n\") pod \"e7260b6d-3070-42b1-93cd-9ec29dfa50c3\" (UID: \"e7260b6d-3070-42b1-93cd-9ec29dfa50c3\") " Oct 11 10:27:05.740678 master-1 kubenswrapper[4771]: I1011 10:27:05.740602 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7260b6d-3070-42b1-93cd-9ec29dfa50c3-kube-api-access-rw99n" (OuterVolumeSpecName: "kube-api-access-rw99n") pod "e7260b6d-3070-42b1-93cd-9ec29dfa50c3" (UID: "e7260b6d-3070-42b1-93cd-9ec29dfa50c3"). InnerVolumeSpecName "kube-api-access-rw99n". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:27:05.835840 master-1 kubenswrapper[4771]: I1011 10:27:05.835714 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-rw99n\" (UniqueName: \"kubernetes.io/projected/e7260b6d-3070-42b1-93cd-9ec29dfa50c3-kube-api-access-rw99n\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:06.611654 master-1 kubenswrapper[4771]: I1011 10:27:06.611589 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/mtu-prober-t9bz7" event={"ID":"e7260b6d-3070-42b1-93cd-9ec29dfa50c3","Type":"ContainerDied","Data":"a6818c7d94ddd80836f979da47140d38cbf5ace31adebfef26025d803442a472"} Oct 11 10:27:06.611987 master-1 kubenswrapper[4771]: I1011 10:27:06.611960 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6818c7d94ddd80836f979da47140d38cbf5ace31adebfef26025d803442a472" Oct 11 10:27:06.612113 master-1 kubenswrapper[4771]: I1011 10:27:06.611677 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/mtu-prober-t9bz7" Oct 11 10:27:08.140298 master-1 kubenswrapper[4771]: I1011 10:27:08.140228 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-network-operator/mtu-prober-t9bz7"] Oct 11 10:27:08.144438 master-1 kubenswrapper[4771]: I1011 10:27:08.144394 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-network-operator/mtu-prober-t9bz7"] Oct 11 10:27:08.441727 master-1 kubenswrapper[4771]: I1011 10:27:08.441581 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7260b6d-3070-42b1-93cd-9ec29dfa50c3" path="/var/lib/kubelet/pods/e7260b6d-3070-42b1-93cd-9ec29dfa50c3/volumes" Oct 11 10:27:13.003239 master-1 kubenswrapper[4771]: I1011 10:27:13.002914 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-dgt7f"] Oct 11 10:27:13.003239 master-1 kubenswrapper[4771]: E1011 10:27:13.003073 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e7260b6d-3070-42b1-93cd-9ec29dfa50c3" containerName="prober" Oct 11 10:27:13.003239 master-1 kubenswrapper[4771]: I1011 10:27:13.003101 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7260b6d-3070-42b1-93cd-9ec29dfa50c3" containerName="prober" Oct 11 10:27:13.003239 master-1 kubenswrapper[4771]: I1011 10:27:13.003138 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e7260b6d-3070-42b1-93cd-9ec29dfa50c3" containerName="prober" Oct 11 10:27:13.004685 master-1 kubenswrapper[4771]: I1011 10:27:13.003589 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.008285 master-1 kubenswrapper[4771]: I1011 10:27:13.008206 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-multus"/"multus-daemon-config" Oct 11 10:27:13.008553 master-1 kubenswrapper[4771]: I1011 10:27:13.008483 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-multus"/"openshift-service-ca.crt" Oct 11 10:27:13.008652 master-1 kubenswrapper[4771]: I1011 10:27:13.008563 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-multus"/"cni-copy-resources" Oct 11 10:27:13.009385 master-1 kubenswrapper[4771]: I1011 10:27:13.009168 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-multus"/"kube-root-ca.crt" Oct 11 10:27:13.182259 master-1 kubenswrapper[4771]: I1011 10:27:13.182098 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-var-lib-cni-multus\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.182259 master-1 kubenswrapper[4771]: I1011 10:27:13.182228 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-run-multus-certs\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.182745 master-1 kubenswrapper[4771]: I1011 10:27:13.182284 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-conf-dir\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.182745 master-1 kubenswrapper[4771]: I1011 10:27:13.182338 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b771285f-4d3c-4a7a-9b62-eb804911a351-cni-binary-copy\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.182745 master-1 kubenswrapper[4771]: I1011 10:27:13.182429 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67k7\" (UniqueName: \"kubernetes.io/projected/b771285f-4d3c-4a7a-9b62-eb804911a351-kube-api-access-r67k7\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.182745 master-1 kubenswrapper[4771]: I1011 10:27:13.182560 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-cni-dir\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.182745 master-1 kubenswrapper[4771]: I1011 10:27:13.182652 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-socket-dir-parent\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.182745 master-1 kubenswrapper[4771]: I1011 10:27:13.182699 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-run-k8s-cni-cncf-io\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.182745 master-1 kubenswrapper[4771]: I1011 10:27:13.182742 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-etc-kubernetes\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.183341 master-1 kubenswrapper[4771]: I1011 10:27:13.182789 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-cnibin\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.183341 master-1 kubenswrapper[4771]: I1011 10:27:13.182836 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-run-netns\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.183341 master-1 kubenswrapper[4771]: I1011 10:27:13.182883 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-system-cni-dir\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.183341 master-1 kubenswrapper[4771]: I1011 10:27:13.182924 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-hostroot\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.183341 master-1 kubenswrapper[4771]: I1011 10:27:13.182971 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-os-release\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.183341 master-1 kubenswrapper[4771]: I1011 10:27:13.183015 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-var-lib-cni-bin\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.183341 master-1 kubenswrapper[4771]: I1011 10:27:13.183065 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-var-lib-kubelet\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.183341 master-1 kubenswrapper[4771]: I1011 10:27:13.183152 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-daemon-config\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.201186 master-1 kubenswrapper[4771]: I1011 10:27:13.201112 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-lvp6f"] Oct 11 10:27:13.202187 master-1 kubenswrapper[4771]: I1011 10:27:13.202111 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.206608 master-1 kubenswrapper[4771]: W1011 10:27:13.206555 4771 reflector.go:561] object-"openshift-multus"/"whereabouts-config": failed to list *v1.ConfigMap: configmaps "whereabouts-config" is forbidden: User "system:node:master-1" cannot list resource "configmaps" in API group "" in the namespace "openshift-multus": no relationship found between node 'master-1' and this object Oct 11 10:27:13.206608 master-1 kubenswrapper[4771]: E1011 10:27:13.206597 4771 reflector.go:158] "Unhandled Error" err="object-\"openshift-multus\"/\"whereabouts-config\": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps \"whereabouts-config\" is forbidden: User \"system:node:master-1\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-multus\": no relationship found between node 'master-1' and this object" logger="UnhandledError" Oct 11 10:27:13.206842 master-1 kubenswrapper[4771]: W1011 10:27:13.206603 4771 reflector.go:561] object-"openshift-multus"/"default-cni-sysctl-allowlist": failed to list *v1.ConfigMap: configmaps "default-cni-sysctl-allowlist" is forbidden: User "system:node:master-1" cannot list resource "configmaps" in API group "" in the namespace "openshift-multus": no relationship found between node 'master-1' and this object Oct 11 10:27:13.206842 master-1 kubenswrapper[4771]: E1011 10:27:13.206657 4771 reflector.go:158] "Unhandled Error" err="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps \"default-cni-sysctl-allowlist\" is forbidden: User \"system:node:master-1\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-multus\": no relationship found between node 'master-1' and this object" logger="UnhandledError" Oct 11 10:27:13.284624 master-1 kubenswrapper[4771]: I1011 10:27:13.284394 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-conf-dir\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.284624 master-1 kubenswrapper[4771]: I1011 10:27:13.284485 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b771285f-4d3c-4a7a-9b62-eb804911a351-cni-binary-copy\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.284624 master-1 kubenswrapper[4771]: I1011 10:27:13.284517 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-r67k7\" (UniqueName: \"kubernetes.io/projected/b771285f-4d3c-4a7a-9b62-eb804911a351-kube-api-access-r67k7\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.284624 master-1 kubenswrapper[4771]: I1011 10:27:13.284541 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-cni-dir\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.284624 master-1 kubenswrapper[4771]: I1011 10:27:13.284547 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-conf-dir\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.284680 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-socket-dir-parent\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.284571 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-socket-dir-parent\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.284862 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-run-k8s-cni-cncf-io\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.284891 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-cni-dir\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.284913 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-etc-kubernetes\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.284956 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-run-netns\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.284975 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-run-k8s-cni-cncf-io\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.284987 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-cnibin\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.285054 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-run-netns\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.285081 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-etc-kubernetes\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.285113 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-system-cni-dir\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.285157 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-cnibin\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.285194 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-hostroot\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.285211 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-system-cni-dir\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.285209 master-1 kubenswrapper[4771]: I1011 10:27:13.285242 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-os-release\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285272 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-hostroot\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285292 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-var-lib-cni-bin\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285393 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-var-lib-cni-bin\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285409 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-os-release\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285437 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-var-lib-kubelet\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285501 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-var-lib-kubelet\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285508 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-daemon-config\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285566 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-var-lib-cni-multus\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285616 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-run-multus-certs\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285661 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-var-lib-cni-multus\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.285714 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b771285f-4d3c-4a7a-9b62-eb804911a351-host-run-multus-certs\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.286500 master-1 kubenswrapper[4771]: I1011 10:27:13.286136 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b771285f-4d3c-4a7a-9b62-eb804911a351-cni-binary-copy\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.287254 master-1 kubenswrapper[4771]: I1011 10:27:13.286679 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b771285f-4d3c-4a7a-9b62-eb804911a351-multus-daemon-config\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.317084 master-1 kubenswrapper[4771]: I1011 10:27:13.316977 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67k7\" (UniqueName: \"kubernetes.io/projected/b771285f-4d3c-4a7a-9b62-eb804911a351-kube-api-access-r67k7\") pod \"multus-dgt7f\" (UID: \"b771285f-4d3c-4a7a-9b62-eb804911a351\") " pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.325939 master-1 kubenswrapper[4771]: I1011 10:27:13.325864 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dgt7f" Oct 11 10:27:13.340816 master-1 kubenswrapper[4771]: W1011 10:27:13.340753 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb771285f_4d3c_4a7a_9b62_eb804911a351.slice/crio-3efe5f729cca6f9537557a566e71fdf1ab0d7f1e220def443492089756528391 WatchSource:0}: Error finding container 3efe5f729cca6f9537557a566e71fdf1ab0d7f1e220def443492089756528391: Status 404 returned error can't find the container with id 3efe5f729cca6f9537557a566e71fdf1ab0d7f1e220def443492089756528391 Oct 11 10:27:13.386847 master-1 kubenswrapper[4771]: I1011 10:27:13.386754 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-system-cni-dir\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.386847 master-1 kubenswrapper[4771]: I1011 10:27:13.386832 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.387050 master-1 kubenswrapper[4771]: I1011 10:27:13.386968 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b4dff81-4eaa-422f-8de9-d6133a8b2016-cni-binary-copy\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.387151 master-1 kubenswrapper[4771]: I1011 10:27:13.387095 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-configmap\" (UniqueName: \"kubernetes.io/configmap/0b4dff81-4eaa-422f-8de9-d6133a8b2016-whereabouts-configmap\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.387216 master-1 kubenswrapper[4771]: I1011 10:27:13.387162 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5m7z\" (UniqueName: \"kubernetes.io/projected/0b4dff81-4eaa-422f-8de9-d6133a8b2016-kube-api-access-t5m7z\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.387216 master-1 kubenswrapper[4771]: I1011 10:27:13.387202 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-os-release\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.387333 master-1 kubenswrapper[4771]: I1011 10:27:13.387237 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b4dff81-4eaa-422f-8de9-d6133a8b2016-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.387333 master-1 kubenswrapper[4771]: I1011 10:27:13.387320 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-cnibin\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.436891 master-1 kubenswrapper[4771]: I1011 10:27:13.436857 4771 scope.go:117] "RemoveContainer" containerID="3fea68ffee0d3f0d9bbafc91305f308f57d148b9eb031b738b0ded06753f61e1" Oct 11 10:27:13.488388 master-1 kubenswrapper[4771]: I1011 10:27:13.488303 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-os-release\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488507 master-1 kubenswrapper[4771]: I1011 10:27:13.488412 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b4dff81-4eaa-422f-8de9-d6133a8b2016-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488507 master-1 kubenswrapper[4771]: I1011 10:27:13.488461 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-cnibin\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488638 master-1 kubenswrapper[4771]: I1011 10:27:13.488514 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-system-cni-dir\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488638 master-1 kubenswrapper[4771]: I1011 10:27:13.488560 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488638 master-1 kubenswrapper[4771]: I1011 10:27:13.488599 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b4dff81-4eaa-422f-8de9-d6133a8b2016-cni-binary-copy\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488638 master-1 kubenswrapper[4771]: I1011 10:27:13.488603 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-os-release\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488857 master-1 kubenswrapper[4771]: I1011 10:27:13.488666 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-cnibin\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488857 master-1 kubenswrapper[4771]: I1011 10:27:13.488636 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"whereabouts-configmap\" (UniqueName: \"kubernetes.io/configmap/0b4dff81-4eaa-422f-8de9-d6133a8b2016-whereabouts-configmap\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488857 master-1 kubenswrapper[4771]: I1011 10:27:13.488710 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-system-cni-dir\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.488857 master-1 kubenswrapper[4771]: I1011 10:27:13.488758 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-t5m7z\" (UniqueName: \"kubernetes.io/projected/0b4dff81-4eaa-422f-8de9-d6133a8b2016-kube-api-access-t5m7z\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.489076 master-1 kubenswrapper[4771]: I1011 10:27:13.488906 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b4dff81-4eaa-422f-8de9-d6133a8b2016-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.489975 master-1 kubenswrapper[4771]: I1011 10:27:13.489921 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b4dff81-4eaa-422f-8de9-d6133a8b2016-cni-binary-copy\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.520224 master-1 kubenswrapper[4771]: I1011 10:27:13.520127 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5m7z\" (UniqueName: \"kubernetes.io/projected/0b4dff81-4eaa-422f-8de9-d6133a8b2016-kube-api-access-t5m7z\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:13.632099 master-1 kubenswrapper[4771]: I1011 10:27:13.632028 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dgt7f" event={"ID":"b771285f-4d3c-4a7a-9b62-eb804911a351","Type":"ContainerStarted","Data":"3efe5f729cca6f9537557a566e71fdf1ab0d7f1e220def443492089756528391"} Oct 11 10:27:13.989043 master-1 kubenswrapper[4771]: I1011 10:27:13.988572 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-fgjvw"] Oct 11 10:27:13.989390 master-1 kubenswrapper[4771]: I1011 10:27:13.989324 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:13.989543 master-1 kubenswrapper[4771]: E1011 10:27:13.989496 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:14.094531 master-1 kubenswrapper[4771]: I1011 10:27:14.094423 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:14.094531 master-1 kubenswrapper[4771]: I1011 10:27:14.094526 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twgg\" (UniqueName: \"kubernetes.io/projected/2c084572-a5c9-4787-8a14-b7d6b0810a1b-kube-api-access-9twgg\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:14.195564 master-1 kubenswrapper[4771]: I1011 10:27:14.195471 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:14.195799 master-1 kubenswrapper[4771]: I1011 10:27:14.195576 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-9twgg\" (UniqueName: \"kubernetes.io/projected/2c084572-a5c9-4787-8a14-b7d6b0810a1b-kube-api-access-9twgg\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:14.195799 master-1 kubenswrapper[4771]: E1011 10:27:14.195738 4771 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:14.195913 master-1 kubenswrapper[4771]: E1011 10:27:14.195852 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs podName:2c084572-a5c9-4787-8a14-b7d6b0810a1b nodeName:}" failed. No retries permitted until 2025-10-11 10:27:14.695825369 +0000 UTC m=+66.670051850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs") pod "network-metrics-daemon-fgjvw" (UID: "2c084572-a5c9-4787-8a14-b7d6b0810a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:14.234719 master-1 kubenswrapper[4771]: I1011 10:27:14.234630 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-multus"/"default-cni-sysctl-allowlist" Oct 11 10:27:14.240500 master-1 kubenswrapper[4771]: I1011 10:27:14.240412 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b4dff81-4eaa-422f-8de9-d6133a8b2016-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:14.241501 master-1 kubenswrapper[4771]: I1011 10:27:14.241443 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twgg\" (UniqueName: \"kubernetes.io/projected/2c084572-a5c9-4787-8a14-b7d6b0810a1b-kube-api-access-9twgg\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:14.295544 master-1 kubenswrapper[4771]: I1011 10:27:14.295478 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-multus"/"whereabouts-config" Oct 11 10:27:14.300306 master-1 kubenswrapper[4771]: I1011 10:27:14.300245 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"whereabouts-configmap\" (UniqueName: \"kubernetes.io/configmap/0b4dff81-4eaa-422f-8de9-d6133a8b2016-whereabouts-configmap\") pod \"multus-additional-cni-plugins-lvp6f\" (UID: \"0b4dff81-4eaa-422f-8de9-d6133a8b2016\") " pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:14.422282 master-1 kubenswrapper[4771]: I1011 10:27:14.422165 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" Oct 11 10:27:14.636467 master-1 kubenswrapper[4771]: I1011 10:27:14.636413 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_4ba9953d-1f54-43be-a3ae-121030f1e07b/kube-rbac-proxy/2.log" Oct 11 10:27:14.637377 master-1 kubenswrapper[4771]: I1011 10:27:14.637317 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_4ba9953d-1f54-43be-a3ae-121030f1e07b/kube-rbac-proxy/1.log" Oct 11 10:27:14.638328 master-1 kubenswrapper[4771]: I1011 10:27:14.638284 4771 generic.go:334] "Generic (PLEG): container finished" podID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerID="8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2" exitCode=1 Oct 11 10:27:14.638409 master-1 kubenswrapper[4771]: I1011 10:27:14.638377 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" event={"ID":"4ba9953d-1f54-43be-a3ae-121030f1e07b","Type":"ContainerDied","Data":"8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2"} Oct 11 10:27:14.638446 master-1 kubenswrapper[4771]: I1011 10:27:14.638434 4771 scope.go:117] "RemoveContainer" containerID="3fea68ffee0d3f0d9bbafc91305f308f57d148b9eb031b738b0ded06753f61e1" Oct 11 10:27:14.639214 master-1 kubenswrapper[4771]: I1011 10:27:14.639171 4771 scope.go:117] "RemoveContainer" containerID="8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2" Oct 11 10:27:14.639474 master-1 kubenswrapper[4771]: E1011 10:27:14.639431 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_openshift-cloud-controller-manager-operator(4ba9953d-1f54-43be-a3ae-121030f1e07b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" Oct 11 10:27:14.639781 master-1 kubenswrapper[4771]: I1011 10:27:14.639688 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" event={"ID":"0b4dff81-4eaa-422f-8de9-d6133a8b2016","Type":"ContainerStarted","Data":"4f3a0d0bd0cb0b63fa93b7e91e0a5742f68e970f73492d04d3ca1e5f37e65916"} Oct 11 10:27:14.699312 master-1 kubenswrapper[4771]: I1011 10:27:14.699225 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:14.699471 master-1 kubenswrapper[4771]: E1011 10:27:14.699412 4771 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:14.699542 master-1 kubenswrapper[4771]: E1011 10:27:14.699475 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs podName:2c084572-a5c9-4787-8a14-b7d6b0810a1b nodeName:}" failed. No retries permitted until 2025-10-11 10:27:15.699458675 +0000 UTC m=+67.673685126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs") pod "network-metrics-daemon-fgjvw" (UID: "2c084572-a5c9-4787-8a14-b7d6b0810a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:15.435998 master-1 kubenswrapper[4771]: I1011 10:27:15.435859 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:15.436946 master-1 kubenswrapper[4771]: E1011 10:27:15.436060 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:15.644883 master-1 kubenswrapper[4771]: I1011 10:27:15.644808 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_4ba9953d-1f54-43be-a3ae-121030f1e07b/kube-rbac-proxy/2.log" Oct 11 10:27:15.706916 master-1 kubenswrapper[4771]: I1011 10:27:15.706799 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:15.707060 master-1 kubenswrapper[4771]: E1011 10:27:15.707023 4771 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:15.707213 master-1 kubenswrapper[4771]: E1011 10:27:15.707157 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs podName:2c084572-a5c9-4787-8a14-b7d6b0810a1b nodeName:}" failed. No retries permitted until 2025-10-11 10:27:17.707122286 +0000 UTC m=+69.681348767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs") pod "network-metrics-daemon-fgjvw" (UID: "2c084572-a5c9-4787-8a14-b7d6b0810a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:17.436730 master-1 kubenswrapper[4771]: I1011 10:27:17.436665 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:17.437935 master-1 kubenswrapper[4771]: E1011 10:27:17.436976 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:17.652068 master-1 kubenswrapper[4771]: I1011 10:27:17.651991 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b4dff81-4eaa-422f-8de9-d6133a8b2016" containerID="75355fa584d02989c18f5564dbedfea406415ac8f1958dd2d81970ffd991509e" exitCode=0 Oct 11 10:27:17.652068 master-1 kubenswrapper[4771]: I1011 10:27:17.652046 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" event={"ID":"0b4dff81-4eaa-422f-8de9-d6133a8b2016","Type":"ContainerDied","Data":"75355fa584d02989c18f5564dbedfea406415ac8f1958dd2d81970ffd991509e"} Oct 11 10:27:17.679385 master-1 kubenswrapper[4771]: I1011 10:27:17.679283 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf"] Oct 11 10:27:17.679559 master-1 kubenswrapper[4771]: I1011 10:27:17.679494 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="cluster-cloud-controller-manager" containerID="cri-o://6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725" gracePeriod=30 Oct 11 10:27:17.679666 master-1 kubenswrapper[4771]: I1011 10:27:17.679563 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="config-sync-controllers" containerID="cri-o://ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62" gracePeriod=30 Oct 11 10:27:17.744771 master-1 kubenswrapper[4771]: I1011 10:27:17.744694 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:17.744949 master-1 kubenswrapper[4771]: E1011 10:27:17.744869 4771 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:17.745129 master-1 kubenswrapper[4771]: E1011 10:27:17.745074 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs podName:2c084572-a5c9-4787-8a14-b7d6b0810a1b nodeName:}" failed. No retries permitted until 2025-10-11 10:27:21.745033945 +0000 UTC m=+73.719260426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs") pod "network-metrics-daemon-fgjvw" (UID: "2c084572-a5c9-4787-8a14-b7d6b0810a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:17.821282 master-1 kubenswrapper[4771]: I1011 10:27:17.821211 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_4ba9953d-1f54-43be-a3ae-121030f1e07b/kube-rbac-proxy/2.log" Oct 11 10:27:17.822380 master-1 kubenswrapper[4771]: I1011 10:27:17.822299 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:27:17.946474 master-1 kubenswrapper[4771]: I1011 10:27:17.946332 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ba9953d-1f54-43be-a3ae-121030f1e07b-host-etc-kube\") pod \"4ba9953d-1f54-43be-a3ae-121030f1e07b\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " Oct 11 10:27:17.946474 master-1 kubenswrapper[4771]: I1011 10:27:17.946427 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cloud-controller-manager-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ba9953d-1f54-43be-a3ae-121030f1e07b-cloud-controller-manager-operator-tls\") pod \"4ba9953d-1f54-43be-a3ae-121030f1e07b\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " Oct 11 10:27:17.946474 master-1 kubenswrapper[4771]: I1011 10:27:17.946471 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"images\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-images\") pod \"4ba9953d-1f54-43be-a3ae-121030f1e07b\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " Oct 11 10:27:17.946734 master-1 kubenswrapper[4771]: I1011 10:27:17.946508 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-auth-proxy-config\") pod \"4ba9953d-1f54-43be-a3ae-121030f1e07b\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " Oct 11 10:27:17.946734 master-1 kubenswrapper[4771]: I1011 10:27:17.946498 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/4ba9953d-1f54-43be-a3ae-121030f1e07b-host-etc-kube" (OuterVolumeSpecName: "host-etc-kube") pod "4ba9953d-1f54-43be-a3ae-121030f1e07b" (UID: "4ba9953d-1f54-43be-a3ae-121030f1e07b"). InnerVolumeSpecName "host-etc-kube". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:17.946734 master-1 kubenswrapper[4771]: I1011 10:27:17.946550 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c85t\" (UniqueName: \"kubernetes.io/projected/4ba9953d-1f54-43be-a3ae-121030f1e07b-kube-api-access-4c85t\") pod \"4ba9953d-1f54-43be-a3ae-121030f1e07b\" (UID: \"4ba9953d-1f54-43be-a3ae-121030f1e07b\") " Oct 11 10:27:17.946734 master-1 kubenswrapper[4771]: I1011 10:27:17.946650 4771 reconciler_common.go:293] "Volume detached for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ba9953d-1f54-43be-a3ae-121030f1e07b-host-etc-kube\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:17.947107 master-1 kubenswrapper[4771]: I1011 10:27:17.947075 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-auth-proxy-config" (OuterVolumeSpecName: "auth-proxy-config") pod "4ba9953d-1f54-43be-a3ae-121030f1e07b" (UID: "4ba9953d-1f54-43be-a3ae-121030f1e07b"). InnerVolumeSpecName "auth-proxy-config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:27:17.947655 master-1 kubenswrapper[4771]: I1011 10:27:17.947584 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-images" (OuterVolumeSpecName: "images") pod "4ba9953d-1f54-43be-a3ae-121030f1e07b" (UID: "4ba9953d-1f54-43be-a3ae-121030f1e07b"). InnerVolumeSpecName "images". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:27:17.951934 master-1 kubenswrapper[4771]: I1011 10:27:17.951883 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba9953d-1f54-43be-a3ae-121030f1e07b-kube-api-access-4c85t" (OuterVolumeSpecName: "kube-api-access-4c85t") pod "4ba9953d-1f54-43be-a3ae-121030f1e07b" (UID: "4ba9953d-1f54-43be-a3ae-121030f1e07b"). InnerVolumeSpecName "kube-api-access-4c85t". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:27:17.952707 master-1 kubenswrapper[4771]: I1011 10:27:17.952650 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba9953d-1f54-43be-a3ae-121030f1e07b-cloud-controller-manager-operator-tls" (OuterVolumeSpecName: "cloud-controller-manager-operator-tls") pod "4ba9953d-1f54-43be-a3ae-121030f1e07b" (UID: "4ba9953d-1f54-43be-a3ae-121030f1e07b"). InnerVolumeSpecName "cloud-controller-manager-operator-tls". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:27:18.047491 master-1 kubenswrapper[4771]: I1011 10:27:18.047396 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-4c85t\" (UniqueName: \"kubernetes.io/projected/4ba9953d-1f54-43be-a3ae-121030f1e07b-kube-api-access-4c85t\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:18.047491 master-1 kubenswrapper[4771]: I1011 10:27:18.047468 4771 reconciler_common.go:293] "Volume detached for volume \"cloud-controller-manager-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ba9953d-1f54-43be-a3ae-121030f1e07b-cloud-controller-manager-operator-tls\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:18.047704 master-1 kubenswrapper[4771]: I1011 10:27:18.047507 4771 reconciler_common.go:293] "Volume detached for volume \"images\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-images\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:18.047704 master-1 kubenswrapper[4771]: I1011 10:27:18.047535 4771 reconciler_common.go:293] "Volume detached for volume \"auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/4ba9953d-1f54-43be-a3ae-121030f1e07b-auth-proxy-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:18.656850 master-1 kubenswrapper[4771]: I1011 10:27:18.656758 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf_4ba9953d-1f54-43be-a3ae-121030f1e07b/kube-rbac-proxy/2.log" Oct 11 10:27:18.657946 master-1 kubenswrapper[4771]: I1011 10:27:18.657866 4771 generic.go:334] "Generic (PLEG): container finished" podID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerID="ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62" exitCode=0 Oct 11 10:27:18.657946 master-1 kubenswrapper[4771]: I1011 10:27:18.657899 4771 generic.go:334] "Generic (PLEG): container finished" podID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerID="6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725" exitCode=0 Oct 11 10:27:18.657946 master-1 kubenswrapper[4771]: I1011 10:27:18.657925 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" event={"ID":"4ba9953d-1f54-43be-a3ae-121030f1e07b","Type":"ContainerDied","Data":"ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62"} Oct 11 10:27:18.658153 master-1 kubenswrapper[4771]: I1011 10:27:18.657956 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" event={"ID":"4ba9953d-1f54-43be-a3ae-121030f1e07b","Type":"ContainerDied","Data":"6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725"} Oct 11 10:27:18.658153 master-1 kubenswrapper[4771]: I1011 10:27:18.657974 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" event={"ID":"4ba9953d-1f54-43be-a3ae-121030f1e07b","Type":"ContainerDied","Data":"45041b48c16ec7268bc8f5e7bf6ad631a6a8600e9455c7fcc91bcbcacf5e65a9"} Oct 11 10:27:18.658153 master-1 kubenswrapper[4771]: I1011 10:27:18.657974 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf" Oct 11 10:27:18.658153 master-1 kubenswrapper[4771]: I1011 10:27:18.658023 4771 scope.go:117] "RemoveContainer" containerID="8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2" Oct 11 10:27:18.676732 master-1 kubenswrapper[4771]: I1011 10:27:18.676691 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf"] Oct 11 10:27:18.680087 master-1 kubenswrapper[4771]: I1011 10:27:18.680042 4771 scope.go:117] "RemoveContainer" containerID="ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62" Oct 11 10:27:18.680925 master-1 kubenswrapper[4771]: I1011 10:27:18.680883 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-6d4bdff5b8-kqrjf"] Oct 11 10:27:18.694020 master-1 kubenswrapper[4771]: I1011 10:27:18.693975 4771 scope.go:117] "RemoveContainer" containerID="6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725" Oct 11 10:27:18.706855 master-1 kubenswrapper[4771]: I1011 10:27:18.706823 4771 scope.go:117] "RemoveContainer" containerID="8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2" Oct 11 10:27:18.707932 master-1 kubenswrapper[4771]: E1011 10:27:18.707881 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2\": container with ID starting with 8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2 not found: ID does not exist" containerID="8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2" Oct 11 10:27:18.708058 master-1 kubenswrapper[4771]: I1011 10:27:18.707925 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2"} err="failed to get container status \"8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2\": rpc error: code = NotFound desc = could not find container \"8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2\": container with ID starting with 8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2 not found: ID does not exist" Oct 11 10:27:18.708058 master-1 kubenswrapper[4771]: I1011 10:27:18.707975 4771 scope.go:117] "RemoveContainer" containerID="ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62" Oct 11 10:27:18.708475 master-1 kubenswrapper[4771]: E1011 10:27:18.708438 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62\": container with ID starting with ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62 not found: ID does not exist" containerID="ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62" Oct 11 10:27:18.708595 master-1 kubenswrapper[4771]: I1011 10:27:18.708474 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62"} err="failed to get container status \"ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62\": rpc error: code = NotFound desc = could not find container \"ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62\": container with ID starting with ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62 not found: ID does not exist" Oct 11 10:27:18.708595 master-1 kubenswrapper[4771]: I1011 10:27:18.708497 4771 scope.go:117] "RemoveContainer" containerID="6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725" Oct 11 10:27:18.709005 master-1 kubenswrapper[4771]: E1011 10:27:18.708970 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725\": container with ID starting with 6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725 not found: ID does not exist" containerID="6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725" Oct 11 10:27:18.709096 master-1 kubenswrapper[4771]: I1011 10:27:18.709004 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725"} err="failed to get container status \"6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725\": rpc error: code = NotFound desc = could not find container \"6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725\": container with ID starting with 6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725 not found: ID does not exist" Oct 11 10:27:18.709096 master-1 kubenswrapper[4771]: I1011 10:27:18.709027 4771 scope.go:117] "RemoveContainer" containerID="8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2" Oct 11 10:27:18.709660 master-1 kubenswrapper[4771]: I1011 10:27:18.709477 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2"} err="failed to get container status \"8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2\": rpc error: code = NotFound desc = could not find container \"8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2\": container with ID starting with 8520322e3dea9c9b65488967ddc22fb2da41fdf46714aac23d0adc0f1ca902c2 not found: ID does not exist" Oct 11 10:27:18.709660 master-1 kubenswrapper[4771]: I1011 10:27:18.709525 4771 scope.go:117] "RemoveContainer" containerID="ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62" Oct 11 10:27:18.710049 master-1 kubenswrapper[4771]: I1011 10:27:18.709990 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62"} err="failed to get container status \"ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62\": rpc error: code = NotFound desc = could not find container \"ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62\": container with ID starting with ab37ae9b4c5a5566e7240156eb5f4a7f5948a585bd81d70fb5ce16c5156bba62 not found: ID does not exist" Oct 11 10:27:18.710049 master-1 kubenswrapper[4771]: I1011 10:27:18.710028 4771 scope.go:117] "RemoveContainer" containerID="6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725" Oct 11 10:27:18.710464 master-1 kubenswrapper[4771]: I1011 10:27:18.710432 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725"} err="failed to get container status \"6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725\": rpc error: code = NotFound desc = could not find container \"6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725\": container with ID starting with 6ea8c6bb10eb614aa2e5d02bf9e477a7b7013b28a2734df5ba12d856f6095725 not found: ID does not exist" Oct 11 10:27:18.715046 master-1 kubenswrapper[4771]: I1011 10:27:18.714995 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp"] Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: E1011 10:27:18.715076 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="kube-rbac-proxy" Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: I1011 10:27:18.715091 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="kube-rbac-proxy" Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: E1011 10:27:18.715100 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="kube-rbac-proxy" Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: I1011 10:27:18.715108 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="kube-rbac-proxy" Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: E1011 10:27:18.715116 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="cluster-cloud-controller-manager" Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: I1011 10:27:18.715124 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="cluster-cloud-controller-manager" Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: E1011 10:27:18.715133 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="config-sync-controllers" Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: I1011 10:27:18.715141 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="config-sync-controllers" Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: E1011 10:27:18.715148 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="kube-rbac-proxy" Oct 11 10:27:18.715153 master-1 kubenswrapper[4771]: I1011 10:27:18.715157 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="kube-rbac-proxy" Oct 11 10:27:18.715835 master-1 kubenswrapper[4771]: I1011 10:27:18.715185 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="cluster-cloud-controller-manager" Oct 11 10:27:18.715835 master-1 kubenswrapper[4771]: I1011 10:27:18.715195 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="config-sync-controllers" Oct 11 10:27:18.715835 master-1 kubenswrapper[4771]: I1011 10:27:18.715204 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="kube-rbac-proxy" Oct 11 10:27:18.715835 master-1 kubenswrapper[4771]: I1011 10:27:18.715212 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="kube-rbac-proxy" Oct 11 10:27:18.715835 master-1 kubenswrapper[4771]: I1011 10:27:18.715247 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" containerName="kube-rbac-proxy" Oct 11 10:27:18.715835 master-1 kubenswrapper[4771]: I1011 10:27:18.715497 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.719321 master-1 kubenswrapper[4771]: I1011 10:27:18.719260 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cloud-controller-manager-operator"/"cloud-controller-manager-images" Oct 11 10:27:18.719842 master-1 kubenswrapper[4771]: I1011 10:27:18.719787 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cloud-controller-manager-operator"/"kube-root-ca.crt" Oct 11 10:27:18.720970 master-1 kubenswrapper[4771]: I1011 10:27:18.720490 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cloud-controller-manager-operator"/"openshift-service-ca.crt" Oct 11 10:27:18.720970 master-1 kubenswrapper[4771]: I1011 10:27:18.720088 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-cloud-controller-manager-operator"/"cloud-controller-manager-operator-tls" Oct 11 10:27:18.720970 master-1 kubenswrapper[4771]: I1011 10:27:18.720688 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cloud-controller-manager-operator"/"kube-rbac-proxy" Oct 11 10:27:18.853301 master-1 kubenswrapper[4771]: I1011 10:27:18.853217 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/e115f8be-9e65-4407-8111-568e5ea8ac1b-host-etc-kube\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.853479 master-1 kubenswrapper[4771]: I1011 10:27:18.853313 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"images\" (UniqueName: \"kubernetes.io/configmap/e115f8be-9e65-4407-8111-568e5ea8ac1b-images\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.853479 master-1 kubenswrapper[4771]: I1011 10:27:18.853422 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cloud-controller-manager-operator-tls\" (UniqueName: \"kubernetes.io/secret/e115f8be-9e65-4407-8111-568e5ea8ac1b-cloud-controller-manager-operator-tls\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.853479 master-1 kubenswrapper[4771]: I1011 10:27:18.853463 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5cn6\" (UniqueName: \"kubernetes.io/projected/e115f8be-9e65-4407-8111-568e5ea8ac1b-kube-api-access-h5cn6\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.853613 master-1 kubenswrapper[4771]: I1011 10:27:18.853512 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/e115f8be-9e65-4407-8111-568e5ea8ac1b-auth-proxy-config\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.954548 master-1 kubenswrapper[4771]: I1011 10:27:18.954465 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/e115f8be-9e65-4407-8111-568e5ea8ac1b-host-etc-kube\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.954548 master-1 kubenswrapper[4771]: I1011 10:27:18.954537 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"images\" (UniqueName: \"kubernetes.io/configmap/e115f8be-9e65-4407-8111-568e5ea8ac1b-images\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.954894 master-1 kubenswrapper[4771]: I1011 10:27:18.954567 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cloud-controller-manager-operator-tls\" (UniqueName: \"kubernetes.io/secret/e115f8be-9e65-4407-8111-568e5ea8ac1b-cloud-controller-manager-operator-tls\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.954894 master-1 kubenswrapper[4771]: I1011 10:27:18.954591 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-h5cn6\" (UniqueName: \"kubernetes.io/projected/e115f8be-9e65-4407-8111-568e5ea8ac1b-kube-api-access-h5cn6\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.954894 master-1 kubenswrapper[4771]: I1011 10:27:18.954618 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/e115f8be-9e65-4407-8111-568e5ea8ac1b-auth-proxy-config\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.954894 master-1 kubenswrapper[4771]: I1011 10:27:18.954699 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-etc-kube\" (UniqueName: \"kubernetes.io/host-path/e115f8be-9e65-4407-8111-568e5ea8ac1b-host-etc-kube\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.955659 master-1 kubenswrapper[4771]: I1011 10:27:18.955606 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/e115f8be-9e65-4407-8111-568e5ea8ac1b-auth-proxy-config\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.956100 master-1 kubenswrapper[4771]: I1011 10:27:18.956006 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"images\" (UniqueName: \"kubernetes.io/configmap/e115f8be-9e65-4407-8111-568e5ea8ac1b-images\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.959536 master-1 kubenswrapper[4771]: I1011 10:27:18.959390 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cloud-controller-manager-operator-tls\" (UniqueName: \"kubernetes.io/secret/e115f8be-9e65-4407-8111-568e5ea8ac1b-cloud-controller-manager-operator-tls\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:18.986925 master-1 kubenswrapper[4771]: I1011 10:27:18.986813 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5cn6\" (UniqueName: \"kubernetes.io/projected/e115f8be-9e65-4407-8111-568e5ea8ac1b-kube-api-access-h5cn6\") pod \"cluster-cloud-controller-manager-operator-779749f859-5xxzp\" (UID: \"e115f8be-9e65-4407-8111-568e5ea8ac1b\") " pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:19.037699 master-1 kubenswrapper[4771]: I1011 10:27:19.037623 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" Oct 11 10:27:19.051644 master-1 kubenswrapper[4771]: W1011 10:27:19.051573 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode115f8be_9e65_4407_8111_568e5ea8ac1b.slice/crio-3ad4baabaec7ad253d370491f06ee20607d577a8f41656c712791c090e5f1999 WatchSource:0}: Error finding container 3ad4baabaec7ad253d370491f06ee20607d577a8f41656c712791c090e5f1999: Status 404 returned error can't find the container with id 3ad4baabaec7ad253d370491f06ee20607d577a8f41656c712791c090e5f1999 Oct 11 10:27:19.436150 master-1 kubenswrapper[4771]: I1011 10:27:19.436080 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:19.436309 master-1 kubenswrapper[4771]: E1011 10:27:19.436268 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:19.662722 master-1 kubenswrapper[4771]: I1011 10:27:19.662661 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" event={"ID":"e115f8be-9e65-4407-8111-568e5ea8ac1b","Type":"ContainerStarted","Data":"57a847f740989cb73678819627757d06b52fb1e138c2a8189da83142b3abbcbc"} Oct 11 10:27:19.662722 master-1 kubenswrapper[4771]: I1011 10:27:19.662710 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" event={"ID":"e115f8be-9e65-4407-8111-568e5ea8ac1b","Type":"ContainerStarted","Data":"42d0299cfe1d4477be84432556c151ce87928a6796608302b2a993479ff1ae79"} Oct 11 10:27:19.662722 master-1 kubenswrapper[4771]: I1011 10:27:19.662721 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" event={"ID":"e115f8be-9e65-4407-8111-568e5ea8ac1b","Type":"ContainerStarted","Data":"3ad4baabaec7ad253d370491f06ee20607d577a8f41656c712791c090e5f1999"} Oct 11 10:27:20.440214 master-1 kubenswrapper[4771]: I1011 10:27:20.440121 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba9953d-1f54-43be-a3ae-121030f1e07b" path="/var/lib/kubelet/pods/4ba9953d-1f54-43be-a3ae-121030f1e07b/volumes" Oct 11 10:27:20.667969 master-1 kubenswrapper[4771]: I1011 10:27:20.667923 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/0.log" Oct 11 10:27:20.668882 master-1 kubenswrapper[4771]: I1011 10:27:20.668833 4771 generic.go:334] "Generic (PLEG): container finished" podID="e115f8be-9e65-4407-8111-568e5ea8ac1b" containerID="5585d78883912bb8eeedc837fe074ce0bf4bdc8294ad85bf3cadcef69368c941" exitCode=1 Oct 11 10:27:20.668924 master-1 kubenswrapper[4771]: I1011 10:27:20.668887 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" event={"ID":"e115f8be-9e65-4407-8111-568e5ea8ac1b","Type":"ContainerDied","Data":"5585d78883912bb8eeedc837fe074ce0bf4bdc8294ad85bf3cadcef69368c941"} Oct 11 10:27:20.669320 master-1 kubenswrapper[4771]: I1011 10:27:20.669288 4771 scope.go:117] "RemoveContainer" containerID="5585d78883912bb8eeedc837fe074ce0bf4bdc8294ad85bf3cadcef69368c941" Oct 11 10:27:21.437053 master-1 kubenswrapper[4771]: I1011 10:27:21.436954 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:21.437467 master-1 kubenswrapper[4771]: E1011 10:27:21.437255 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:21.800151 master-1 kubenswrapper[4771]: I1011 10:27:21.799981 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:21.800832 master-1 kubenswrapper[4771]: E1011 10:27:21.800229 4771 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:21.800832 master-1 kubenswrapper[4771]: E1011 10:27:21.800336 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs podName:2c084572-a5c9-4787-8a14-b7d6b0810a1b nodeName:}" failed. No retries permitted until 2025-10-11 10:27:29.80030183 +0000 UTC m=+81.774528311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs") pod "network-metrics-daemon-fgjvw" (UID: "2c084572-a5c9-4787-8a14-b7d6b0810a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:23.435930 master-1 kubenswrapper[4771]: I1011 10:27:23.435865 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:23.436586 master-1 kubenswrapper[4771]: E1011 10:27:23.436101 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:25.388993 master-1 kubenswrapper[4771]: I1011 10:27:25.388874 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb"] Oct 11 10:27:25.389905 master-1 kubenswrapper[4771]: I1011 10:27:25.389250 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.391985 master-1 kubenswrapper[4771]: I1011 10:27:25.391906 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-ovn-kubernetes"/"ovn-control-plane-metrics-cert" Oct 11 10:27:25.392142 master-1 kubenswrapper[4771]: I1011 10:27:25.392097 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ovn-kubernetes"/"kube-root-ca.crt" Oct 11 10:27:25.392640 master-1 kubenswrapper[4771]: I1011 10:27:25.392522 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ovn-kubernetes"/"openshift-service-ca.crt" Oct 11 10:27:25.392640 master-1 kubenswrapper[4771]: I1011 10:27:25.392524 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ovn-kubernetes"/"env-overrides" Oct 11 10:27:25.393352 master-1 kubenswrapper[4771]: I1011 10:27:25.393288 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ovn-kubernetes"/"ovnkube-config" Oct 11 10:27:25.426483 master-1 kubenswrapper[4771]: I1011 10:27:25.426395 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-env-overrides\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.426483 master-1 kubenswrapper[4771]: I1011 10:27:25.426474 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snph7\" (UniqueName: \"kubernetes.io/projected/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-kube-api-access-snph7\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.426716 master-1 kubenswrapper[4771]: I1011 10:27:25.426519 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-ovnkube-config\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.426716 master-1 kubenswrapper[4771]: I1011 10:27:25.426555 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-control-plane-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-ovn-control-plane-metrics-cert\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.436874 master-1 kubenswrapper[4771]: I1011 10:27:25.436756 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:25.437085 master-1 kubenswrapper[4771]: E1011 10:27:25.437018 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:25.527262 master-1 kubenswrapper[4771]: I1011 10:27:25.527179 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-env-overrides\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.527262 master-1 kubenswrapper[4771]: I1011 10:27:25.527242 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-snph7\" (UniqueName: \"kubernetes.io/projected/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-kube-api-access-snph7\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.527556 master-1 kubenswrapper[4771]: I1011 10:27:25.527282 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-ovnkube-config\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.527556 master-1 kubenswrapper[4771]: I1011 10:27:25.527319 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovn-control-plane-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-ovn-control-plane-metrics-cert\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.528321 master-1 kubenswrapper[4771]: I1011 10:27:25.528250 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-env-overrides\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.528787 master-1 kubenswrapper[4771]: I1011 10:27:25.528726 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-ovnkube-config\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.532869 master-1 kubenswrapper[4771]: I1011 10:27:25.532819 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovn-control-plane-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-ovn-control-plane-metrics-cert\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.546934 master-1 kubenswrapper[4771]: I1011 10:27:25.546820 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-snph7\" (UniqueName: \"kubernetes.io/projected/a65a56b0-5ee8-4429-8fe5-b33a6f29bc79-kube-api-access-snph7\") pod \"ovnkube-control-plane-864d695c77-5mflb\" (UID: \"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79\") " pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.594378 master-1 kubenswrapper[4771]: I1011 10:27:25.594240 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-fl2bs"] Oct 11 10:27:25.595719 master-1 kubenswrapper[4771]: I1011 10:27:25.595668 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.598580 master-1 kubenswrapper[4771]: I1011 10:27:25.598530 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-ovn-kubernetes"/"ovn-node-metrics-cert" Oct 11 10:27:25.598733 master-1 kubenswrapper[4771]: I1011 10:27:25.598681 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ovn-kubernetes"/"ovnkube-script-lib" Oct 11 10:27:25.628383 master-1 kubenswrapper[4771]: I1011 10:27:25.628290 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96c2d0f1-e436-480c-9e34-9068178f9df4-ovn-node-metrics-cert\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.628504 master-1 kubenswrapper[4771]: I1011 10:27:25.628445 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-netns\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.628572 master-1 kubenswrapper[4771]: I1011 10:27:25.628502 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-var-lib-openvswitch\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.628572 master-1 kubenswrapper[4771]: I1011 10:27:25.628553 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-netd\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.628752 master-1 kubenswrapper[4771]: I1011 10:27:25.628651 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-slash\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.628752 master-1 kubenswrapper[4771]: I1011 10:27:25.628707 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-ovn-kubernetes\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.628874 master-1 kubenswrapper[4771]: I1011 10:27:25.628781 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-config\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.628931 master-1 kubenswrapper[4771]: I1011 10:27:25.628867 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-systemd-units\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.628931 master-1 kubenswrapper[4771]: I1011 10:27:25.628913 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-node-log\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629043 master-1 kubenswrapper[4771]: I1011 10:27:25.628961 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-systemd\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629043 master-1 kubenswrapper[4771]: I1011 10:27:25.629003 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-script-lib\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629161 master-1 kubenswrapper[4771]: I1011 10:27:25.629101 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-etc-openvswitch\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629224 master-1 kubenswrapper[4771]: I1011 10:27:25.629170 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-ovn\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629287 master-1 kubenswrapper[4771]: I1011 10:27:25.629202 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42mmb\" (UniqueName: \"kubernetes.io/projected/96c2d0f1-e436-480c-9e34-9068178f9df4-kube-api-access-42mmb\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629287 master-1 kubenswrapper[4771]: I1011 10:27:25.629254 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-log-socket\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629445 master-1 kubenswrapper[4771]: I1011 10:27:25.629278 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-bin\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629445 master-1 kubenswrapper[4771]: I1011 10:27:25.629320 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-kubelet\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629445 master-1 kubenswrapper[4771]: I1011 10:27:25.629341 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-openvswitch\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629445 master-1 kubenswrapper[4771]: I1011 10:27:25.629400 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-env-overrides\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.629445 master-1 kubenswrapper[4771]: I1011 10:27:25.629426 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.685732 master-1 kubenswrapper[4771]: I1011 10:27:25.685574 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dgt7f" event={"ID":"b771285f-4d3c-4a7a-9b62-eb804911a351","Type":"ContainerStarted","Data":"2ae231d265f99ca0d5be6ba2301ffc9f7494c6d013c892b6b113832beeac7c6a"} Oct 11 10:27:25.688422 master-1 kubenswrapper[4771]: I1011 10:27:25.688328 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b4dff81-4eaa-422f-8de9-d6133a8b2016" containerID="e43fead2c3261f1d78ff1e7b98f5daaa5f216da5d07993d10eb96f3476c0730b" exitCode=0 Oct 11 10:27:25.688502 master-1 kubenswrapper[4771]: I1011 10:27:25.688437 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" event={"ID":"0b4dff81-4eaa-422f-8de9-d6133a8b2016","Type":"ContainerDied","Data":"e43fead2c3261f1d78ff1e7b98f5daaa5f216da5d07993d10eb96f3476c0730b"} Oct 11 10:27:25.690775 master-1 kubenswrapper[4771]: I1011 10:27:25.690195 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/1.log" Oct 11 10:27:25.691011 master-1 kubenswrapper[4771]: I1011 10:27:25.690951 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/0.log" Oct 11 10:27:25.692065 master-1 kubenswrapper[4771]: I1011 10:27:25.691957 4771 generic.go:334] "Generic (PLEG): container finished" podID="e115f8be-9e65-4407-8111-568e5ea8ac1b" containerID="60a93fb0643f4c652f352528fa2f68ed301ee9978f9f9a0561174149a6bb77e4" exitCode=1 Oct 11 10:27:25.692065 master-1 kubenswrapper[4771]: I1011 10:27:25.692013 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" event={"ID":"e115f8be-9e65-4407-8111-568e5ea8ac1b","Type":"ContainerDied","Data":"60a93fb0643f4c652f352528fa2f68ed301ee9978f9f9a0561174149a6bb77e4"} Oct 11 10:27:25.692065 master-1 kubenswrapper[4771]: I1011 10:27:25.692070 4771 scope.go:117] "RemoveContainer" containerID="5585d78883912bb8eeedc837fe074ce0bf4bdc8294ad85bf3cadcef69368c941" Oct 11 10:27:25.692935 master-1 kubenswrapper[4771]: I1011 10:27:25.692653 4771 scope.go:117] "RemoveContainer" containerID="60a93fb0643f4c652f352528fa2f68ed301ee9978f9f9a0561174149a6bb77e4" Oct 11 10:27:25.692935 master-1 kubenswrapper[4771]: E1011 10:27:25.692898 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-779749f859-5xxzp_openshift-cloud-controller-manager-operator(e115f8be-9e65-4407-8111-568e5ea8ac1b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" podUID="e115f8be-9e65-4407-8111-568e5ea8ac1b" Oct 11 10:27:25.708229 master-1 kubenswrapper[4771]: I1011 10:27:25.708162 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" Oct 11 10:27:25.722856 master-1 kubenswrapper[4771]: I1011 10:27:25.722765 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-dgt7f" podStartSLOduration=2.33034688 podStartE2EDuration="13.72274408s" podCreationTimestamp="2025-10-11 10:27:12 +0000 UTC" firstStartedPulling="2025-10-11 10:27:13.344476928 +0000 UTC m=+65.318703399" lastFinishedPulling="2025-10-11 10:27:24.736874158 +0000 UTC m=+76.711100599" observedRunningTime="2025-10-11 10:27:25.706126749 +0000 UTC m=+77.680353270" watchObservedRunningTime="2025-10-11 10:27:25.72274408 +0000 UTC m=+77.696970531" Oct 11 10:27:25.724343 master-1 kubenswrapper[4771]: W1011 10:27:25.724235 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65a56b0_5ee8_4429_8fe5_b33a6f29bc79.slice/crio-594577b822dd9370218d1cfefe2fcdad44b1be28decab023f29fc49ccb33659e WatchSource:0}: Error finding container 594577b822dd9370218d1cfefe2fcdad44b1be28decab023f29fc49ccb33659e: Status 404 returned error can't find the container with id 594577b822dd9370218d1cfefe2fcdad44b1be28decab023f29fc49ccb33659e Oct 11 10:27:25.729897 master-1 kubenswrapper[4771]: I1011 10:27:25.729823 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-etc-openvswitch\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.729986 master-1 kubenswrapper[4771]: I1011 10:27:25.729920 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-ovn\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.729986 master-1 kubenswrapper[4771]: I1011 10:27:25.729929 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-etc-openvswitch\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.729986 master-1 kubenswrapper[4771]: I1011 10:27:25.729973 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-42mmb\" (UniqueName: \"kubernetes.io/projected/96c2d0f1-e436-480c-9e34-9068178f9df4-kube-api-access-42mmb\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730236 master-1 kubenswrapper[4771]: I1011 10:27:25.730010 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-ovn\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730236 master-1 kubenswrapper[4771]: I1011 10:27:25.730080 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-log-socket\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730236 master-1 kubenswrapper[4771]: I1011 10:27:25.730129 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-bin\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730236 master-1 kubenswrapper[4771]: I1011 10:27:25.730176 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-kubelet\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730236 master-1 kubenswrapper[4771]: I1011 10:27:25.730227 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-openvswitch\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730629 master-1 kubenswrapper[4771]: I1011 10:27:25.730277 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-bin\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730629 master-1 kubenswrapper[4771]: I1011 10:27:25.730311 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-log-socket\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730629 master-1 kubenswrapper[4771]: I1011 10:27:25.730407 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-kubelet\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730629 master-1 kubenswrapper[4771]: I1011 10:27:25.730285 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-env-overrides\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730629 master-1 kubenswrapper[4771]: I1011 10:27:25.730467 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-openvswitch\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730629 master-1 kubenswrapper[4771]: I1011 10:27:25.730533 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730984 master-1 kubenswrapper[4771]: I1011 10:27:25.730648 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96c2d0f1-e436-480c-9e34-9068178f9df4-ovn-node-metrics-cert\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730984 master-1 kubenswrapper[4771]: I1011 10:27:25.730698 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-netns\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730984 master-1 kubenswrapper[4771]: I1011 10:27:25.730775 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-var-lib-openvswitch\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730984 master-1 kubenswrapper[4771]: I1011 10:27:25.730811 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-netd\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730984 master-1 kubenswrapper[4771]: I1011 10:27:25.730928 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-slash\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.730984 master-1 kubenswrapper[4771]: I1011 10:27:25.730968 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-ovn-kubernetes\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731304 master-1 kubenswrapper[4771]: I1011 10:27:25.730986 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-netd\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731304 master-1 kubenswrapper[4771]: I1011 10:27:25.731045 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-netns\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731304 master-1 kubenswrapper[4771]: I1011 10:27:25.731045 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-config\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731304 master-1 kubenswrapper[4771]: I1011 10:27:25.731171 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-systemd-units\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731304 master-1 kubenswrapper[4771]: I1011 10:27:25.731205 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-node-log\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731304 master-1 kubenswrapper[4771]: I1011 10:27:25.731237 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-systemd\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731304 master-1 kubenswrapper[4771]: I1011 10:27:25.731268 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-script-lib\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731730 master-1 kubenswrapper[4771]: I1011 10:27:25.731451 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-env-overrides\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731730 master-1 kubenswrapper[4771]: I1011 10:27:25.731570 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-slash\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731730 master-1 kubenswrapper[4771]: I1011 10:27:25.731700 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-ovn-kubernetes\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731900 master-1 kubenswrapper[4771]: I1011 10:27:25.730930 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731900 master-1 kubenswrapper[4771]: I1011 10:27:25.731835 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-node-log\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.731900 master-1 kubenswrapper[4771]: I1011 10:27:25.731896 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-systemd-units\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.732426 master-1 kubenswrapper[4771]: I1011 10:27:25.732175 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-config\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.732426 master-1 kubenswrapper[4771]: I1011 10:27:25.732221 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-var-lib-openvswitch\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.732426 master-1 kubenswrapper[4771]: I1011 10:27:25.732308 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-systemd\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.732847 master-1 kubenswrapper[4771]: I1011 10:27:25.732782 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-script-lib\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.734333 master-1 kubenswrapper[4771]: I1011 10:27:25.734277 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96c2d0f1-e436-480c-9e34-9068178f9df4-ovn-node-metrics-cert\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.750140 master-1 kubenswrapper[4771]: I1011 10:27:25.750069 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-42mmb\" (UniqueName: \"kubernetes.io/projected/96c2d0f1-e436-480c-9e34-9068178f9df4-kube-api-access-42mmb\") pod \"ovnkube-node-fl2bs\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:25.915904 master-1 kubenswrapper[4771]: I1011 10:27:25.915780 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:26.699042 master-1 kubenswrapper[4771]: I1011 10:27:26.698510 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/1.log" Oct 11 10:27:26.711057 master-1 kubenswrapper[4771]: I1011 10:27:26.701282 4771 scope.go:117] "RemoveContainer" containerID="60a93fb0643f4c652f352528fa2f68ed301ee9978f9f9a0561174149a6bb77e4" Oct 11 10:27:26.711057 master-1 kubenswrapper[4771]: E1011 10:27:26.701564 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-779749f859-5xxzp_openshift-cloud-controller-manager-operator(e115f8be-9e65-4407-8111-568e5ea8ac1b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" podUID="e115f8be-9e65-4407-8111-568e5ea8ac1b" Oct 11 10:27:26.711057 master-1 kubenswrapper[4771]: I1011 10:27:26.702245 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerStarted","Data":"e331eefac7c84cea9d904666f2942a1490b780d5057ea46b8fe92374c1ddb75a"} Oct 11 10:27:26.711057 master-1 kubenswrapper[4771]: I1011 10:27:26.704063 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" event={"ID":"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79","Type":"ContainerStarted","Data":"6943eb663574984dc2c6e6a328d40af44972ef8e57e6085814bb716f16c316dc"} Oct 11 10:27:26.711057 master-1 kubenswrapper[4771]: I1011 10:27:26.704102 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" event={"ID":"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79","Type":"ContainerStarted","Data":"594577b822dd9370218d1cfefe2fcdad44b1be28decab023f29fc49ccb33659e"} Oct 11 10:27:27.436656 master-1 kubenswrapper[4771]: I1011 10:27:27.436591 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:27.436872 master-1 kubenswrapper[4771]: E1011 10:27:27.436814 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:27.710449 master-1 kubenswrapper[4771]: I1011 10:27:27.710179 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b4dff81-4eaa-422f-8de9-d6133a8b2016" containerID="a9cdbd63dbf81d2a97c21e49b0386b8b1e8f6d6a57c9051435ab9ea7049d83ea" exitCode=0 Oct 11 10:27:27.710449 master-1 kubenswrapper[4771]: I1011 10:27:27.710255 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" event={"ID":"0b4dff81-4eaa-422f-8de9-d6133a8b2016","Type":"ContainerDied","Data":"a9cdbd63dbf81d2a97c21e49b0386b8b1e8f6d6a57c9051435ab9ea7049d83ea"} Oct 11 10:27:28.587135 master-1 kubenswrapper[4771]: I1011 10:27:28.587047 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-4pm7x"] Oct 11 10:27:28.587706 master-1 kubenswrapper[4771]: I1011 10:27:28.587670 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:28.587836 master-1 kubenswrapper[4771]: E1011 10:27:28.587797 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:28.656264 master-1 kubenswrapper[4771]: I1011 10:27:28.656208 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:28.757009 master-1 kubenswrapper[4771]: I1011 10:27:28.756964 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:28.775920 master-1 kubenswrapper[4771]: E1011 10:27:28.775850 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 11 10:27:28.775920 master-1 kubenswrapper[4771]: E1011 10:27:28.775896 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 11 10:27:28.775920 master-1 kubenswrapper[4771]: E1011 10:27:28.775914 4771 projected.go:194] Error preparing data for projected volume kube-api-access-hktrh for pod openshift-network-diagnostics/network-check-target-4pm7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:28.776146 master-1 kubenswrapper[4771]: E1011 10:27:28.775975 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh podName:0bde275d-f0a5-4bea-93f7-edd2077e46b4 nodeName:}" failed. No retries permitted until 2025-10-11 10:27:29.275955453 +0000 UTC m=+81.250181894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hktrh" (UniqueName: "kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh") pod "network-check-target-4pm7x" (UID: "0bde275d-f0a5-4bea-93f7-edd2077e46b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:29.361228 master-1 kubenswrapper[4771]: I1011 10:27:29.361131 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:29.361533 master-1 kubenswrapper[4771]: E1011 10:27:29.361379 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 11 10:27:29.361533 master-1 kubenswrapper[4771]: E1011 10:27:29.361407 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 11 10:27:29.361533 master-1 kubenswrapper[4771]: E1011 10:27:29.361424 4771 projected.go:194] Error preparing data for projected volume kube-api-access-hktrh for pod openshift-network-diagnostics/network-check-target-4pm7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:29.361533 master-1 kubenswrapper[4771]: E1011 10:27:29.361520 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh podName:0bde275d-f0a5-4bea-93f7-edd2077e46b4 nodeName:}" failed. No retries permitted until 2025-10-11 10:27:30.361499312 +0000 UTC m=+82.335725773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hktrh" (UniqueName: "kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh") pod "network-check-target-4pm7x" (UID: "0bde275d-f0a5-4bea-93f7-edd2077e46b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:29.436963 master-1 kubenswrapper[4771]: I1011 10:27:29.436849 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:29.437128 master-1 kubenswrapper[4771]: E1011 10:27:29.437045 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:29.719258 master-1 kubenswrapper[4771]: I1011 10:27:29.719121 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b4dff81-4eaa-422f-8de9-d6133a8b2016" containerID="38c544c0979325797108d1727473710ae3aa96ae14dc1bedbecd3ddf365a5f0e" exitCode=0 Oct 11 10:27:29.719258 master-1 kubenswrapper[4771]: I1011 10:27:29.719178 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" event={"ID":"0b4dff81-4eaa-422f-8de9-d6133a8b2016","Type":"ContainerDied","Data":"38c544c0979325797108d1727473710ae3aa96ae14dc1bedbecd3ddf365a5f0e"} Oct 11 10:27:29.865582 master-1 kubenswrapper[4771]: I1011 10:27:29.865536 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:29.866109 master-1 kubenswrapper[4771]: E1011 10:27:29.865753 4771 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:29.866109 master-1 kubenswrapper[4771]: E1011 10:27:29.865867 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs podName:2c084572-a5c9-4787-8a14-b7d6b0810a1b nodeName:}" failed. No retries permitted until 2025-10-11 10:27:45.865838056 +0000 UTC m=+97.840064527 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs") pod "network-metrics-daemon-fgjvw" (UID: "2c084572-a5c9-4787-8a14-b7d6b0810a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:30.368787 master-1 kubenswrapper[4771]: I1011 10:27:30.368731 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:30.369068 master-1 kubenswrapper[4771]: E1011 10:27:30.368899 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 11 10:27:30.369068 master-1 kubenswrapper[4771]: E1011 10:27:30.368948 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 11 10:27:30.369068 master-1 kubenswrapper[4771]: E1011 10:27:30.368964 4771 projected.go:194] Error preparing data for projected volume kube-api-access-hktrh for pod openshift-network-diagnostics/network-check-target-4pm7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:30.369068 master-1 kubenswrapper[4771]: E1011 10:27:30.369032 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh podName:0bde275d-f0a5-4bea-93f7-edd2077e46b4 nodeName:}" failed. No retries permitted until 2025-10-11 10:27:32.369005918 +0000 UTC m=+84.343232359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hktrh" (UniqueName: "kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh") pod "network-check-target-4pm7x" (UID: "0bde275d-f0a5-4bea-93f7-edd2077e46b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:30.436927 master-1 kubenswrapper[4771]: I1011 10:27:30.436872 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:30.437041 master-1 kubenswrapper[4771]: E1011 10:27:30.436972 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:31.211555 master-1 kubenswrapper[4771]: I1011 10:27:31.211499 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-network-node-identity/network-node-identity-sk5cm"] Oct 11 10:27:31.212062 master-1 kubenswrapper[4771]: I1011 10:27:31.211836 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.215379 master-1 kubenswrapper[4771]: I1011 10:27:31.215329 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-node-identity"/"openshift-service-ca.crt" Oct 11 10:27:31.215459 master-1 kubenswrapper[4771]: I1011 10:27:31.215419 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-node-identity"/"kube-root-ca.crt" Oct 11 10:27:31.216217 master-1 kubenswrapper[4771]: I1011 10:27:31.216177 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-node-identity"/"env-overrides" Oct 11 10:27:31.216277 master-1 kubenswrapper[4771]: I1011 10:27:31.216220 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-network-node-identity"/"network-node-identity-cert" Oct 11 10:27:31.216326 master-1 kubenswrapper[4771]: I1011 10:27:31.216300 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-node-identity"/"ovnkube-identity-cm" Oct 11 10:27:31.285387 master-1 kubenswrapper[4771]: I1011 10:27:31.274949 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e03dddd-4197-40ae-91f1-7e83f90dbd58-webhook-cert\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.285387 master-1 kubenswrapper[4771]: I1011 10:27:31.275061 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-identity-cm\" (UniqueName: \"kubernetes.io/configmap/0e03dddd-4197-40ae-91f1-7e83f90dbd58-ovnkube-identity-cm\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.285387 master-1 kubenswrapper[4771]: I1011 10:27:31.275126 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn9ks\" (UniqueName: \"kubernetes.io/projected/0e03dddd-4197-40ae-91f1-7e83f90dbd58-kube-api-access-kn9ks\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.285387 master-1 kubenswrapper[4771]: I1011 10:27:31.275239 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e03dddd-4197-40ae-91f1-7e83f90dbd58-env-overrides\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.376257 master-1 kubenswrapper[4771]: I1011 10:27:31.376191 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e03dddd-4197-40ae-91f1-7e83f90dbd58-webhook-cert\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.376257 master-1 kubenswrapper[4771]: I1011 10:27:31.376242 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovnkube-identity-cm\" (UniqueName: \"kubernetes.io/configmap/0e03dddd-4197-40ae-91f1-7e83f90dbd58-ovnkube-identity-cm\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.376257 master-1 kubenswrapper[4771]: I1011 10:27:31.376275 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-kn9ks\" (UniqueName: \"kubernetes.io/projected/0e03dddd-4197-40ae-91f1-7e83f90dbd58-kube-api-access-kn9ks\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.376620 master-1 kubenswrapper[4771]: I1011 10:27:31.376304 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e03dddd-4197-40ae-91f1-7e83f90dbd58-env-overrides\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.377115 master-1 kubenswrapper[4771]: I1011 10:27:31.377079 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e03dddd-4197-40ae-91f1-7e83f90dbd58-env-overrides\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.377843 master-1 kubenswrapper[4771]: I1011 10:27:31.377791 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovnkube-identity-cm\" (UniqueName: \"kubernetes.io/configmap/0e03dddd-4197-40ae-91f1-7e83f90dbd58-ovnkube-identity-cm\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.380270 master-1 kubenswrapper[4771]: I1011 10:27:31.380207 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e03dddd-4197-40ae-91f1-7e83f90dbd58-webhook-cert\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.394393 master-1 kubenswrapper[4771]: I1011 10:27:31.394340 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn9ks\" (UniqueName: \"kubernetes.io/projected/0e03dddd-4197-40ae-91f1-7e83f90dbd58-kube-api-access-kn9ks\") pod \"network-node-identity-sk5cm\" (UID: \"0e03dddd-4197-40ae-91f1-7e83f90dbd58\") " pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.436106 master-1 kubenswrapper[4771]: I1011 10:27:31.436044 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:31.436235 master-1 kubenswrapper[4771]: E1011 10:27:31.436192 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:31.527158 master-1 kubenswrapper[4771]: I1011 10:27:31.526965 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-node-identity/network-node-identity-sk5cm" Oct 11 10:27:31.541396 master-1 kubenswrapper[4771]: W1011 10:27:31.541328 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e03dddd_4197_40ae_91f1_7e83f90dbd58.slice/crio-0acd29c92f7521b4c20ddb03ce7c41f516fe6d5ba831200b335c125fc0e7499f WatchSource:0}: Error finding container 0acd29c92f7521b4c20ddb03ce7c41f516fe6d5ba831200b335c125fc0e7499f: Status 404 returned error can't find the container with id 0acd29c92f7521b4c20ddb03ce7c41f516fe6d5ba831200b335c125fc0e7499f Oct 11 10:27:31.723712 master-1 kubenswrapper[4771]: I1011 10:27:31.723649 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-node-identity/network-node-identity-sk5cm" event={"ID":"0e03dddd-4197-40ae-91f1-7e83f90dbd58","Type":"ContainerStarted","Data":"0acd29c92f7521b4c20ddb03ce7c41f516fe6d5ba831200b335c125fc0e7499f"} Oct 11 10:27:32.384851 master-1 kubenswrapper[4771]: I1011 10:27:32.384757 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:32.385673 master-1 kubenswrapper[4771]: E1011 10:27:32.384978 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 11 10:27:32.385673 master-1 kubenswrapper[4771]: E1011 10:27:32.385006 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 11 10:27:32.385673 master-1 kubenswrapper[4771]: E1011 10:27:32.385026 4771 projected.go:194] Error preparing data for projected volume kube-api-access-hktrh for pod openshift-network-diagnostics/network-check-target-4pm7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:32.385673 master-1 kubenswrapper[4771]: E1011 10:27:32.385098 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh podName:0bde275d-f0a5-4bea-93f7-edd2077e46b4 nodeName:}" failed. No retries permitted until 2025-10-11 10:27:36.385075078 +0000 UTC m=+88.359301559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hktrh" (UniqueName: "kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh") pod "network-check-target-4pm7x" (UID: "0bde275d-f0a5-4bea-93f7-edd2077e46b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:32.437931 master-1 kubenswrapper[4771]: I1011 10:27:32.437858 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:32.438576 master-1 kubenswrapper[4771]: E1011 10:27:32.438520 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:33.436072 master-1 kubenswrapper[4771]: I1011 10:27:33.435987 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:33.436890 master-1 kubenswrapper[4771]: E1011 10:27:33.436201 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:34.436730 master-1 kubenswrapper[4771]: I1011 10:27:34.436664 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:34.437223 master-1 kubenswrapper[4771]: E1011 10:27:34.436820 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:35.436983 master-1 kubenswrapper[4771]: I1011 10:27:35.436906 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:35.437672 master-1 kubenswrapper[4771]: E1011 10:27:35.437084 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:36.417998 master-1 kubenswrapper[4771]: I1011 10:27:36.417942 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:36.418220 master-1 kubenswrapper[4771]: E1011 10:27:36.418123 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 11 10:27:36.418220 master-1 kubenswrapper[4771]: E1011 10:27:36.418144 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 11 10:27:36.418220 master-1 kubenswrapper[4771]: E1011 10:27:36.418158 4771 projected.go:194] Error preparing data for projected volume kube-api-access-hktrh for pod openshift-network-diagnostics/network-check-target-4pm7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:36.418345 master-1 kubenswrapper[4771]: E1011 10:27:36.418245 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh podName:0bde275d-f0a5-4bea-93f7-edd2077e46b4 nodeName:}" failed. No retries permitted until 2025-10-11 10:27:44.418204486 +0000 UTC m=+96.392430937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hktrh" (UniqueName: "kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh") pod "network-check-target-4pm7x" (UID: "0bde275d-f0a5-4bea-93f7-edd2077e46b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:36.437045 master-1 kubenswrapper[4771]: I1011 10:27:36.437022 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:36.437450 master-1 kubenswrapper[4771]: E1011 10:27:36.437122 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:37.436225 master-1 kubenswrapper[4771]: I1011 10:27:37.436117 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:37.436597 master-1 kubenswrapper[4771]: E1011 10:27:37.436342 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:38.437138 master-1 kubenswrapper[4771]: I1011 10:27:38.436796 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:38.437138 master-1 kubenswrapper[4771]: E1011 10:27:38.437043 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:39.436023 master-1 kubenswrapper[4771]: I1011 10:27:39.435951 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:39.436299 master-1 kubenswrapper[4771]: E1011 10:27:39.436117 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:40.437084 master-1 kubenswrapper[4771]: I1011 10:27:40.436995 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:40.437803 master-1 kubenswrapper[4771]: E1011 10:27:40.437147 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:41.436372 master-1 kubenswrapper[4771]: I1011 10:27:41.436307 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:41.436743 master-1 kubenswrapper[4771]: E1011 10:27:41.436465 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:42.436084 master-1 kubenswrapper[4771]: I1011 10:27:42.435963 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:42.437004 master-1 kubenswrapper[4771]: E1011 10:27:42.436948 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:42.437943 master-1 kubenswrapper[4771]: I1011 10:27:42.437882 4771 scope.go:117] "RemoveContainer" containerID="60a93fb0643f4c652f352528fa2f68ed301ee9978f9f9a0561174149a6bb77e4" Oct 11 10:27:42.756833 master-1 kubenswrapper[4771]: I1011 10:27:42.756766 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-node-identity/network-node-identity-sk5cm" event={"ID":"0e03dddd-4197-40ae-91f1-7e83f90dbd58","Type":"ContainerStarted","Data":"c93dfaf9a8b9fa7850e31e158a74ae1fbf85ec41153c0883cb5064b10872afdb"} Oct 11 10:27:42.756833 master-1 kubenswrapper[4771]: I1011 10:27:42.756832 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-node-identity/network-node-identity-sk5cm" event={"ID":"0e03dddd-4197-40ae-91f1-7e83f90dbd58","Type":"ContainerStarted","Data":"022f8f384bb7270fddd7540b511c399935f2c7dabee6416c3281661a660bc23f"} Oct 11 10:27:42.762620 master-1 kubenswrapper[4771]: I1011 10:27:42.762574 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b4dff81-4eaa-422f-8de9-d6133a8b2016" containerID="e16f8a87364f9124b8fa2ebe666b652656bef4fcb7b7b1b7a185fbeea9bd8939" exitCode=0 Oct 11 10:27:42.762804 master-1 kubenswrapper[4771]: I1011 10:27:42.762754 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" event={"ID":"0b4dff81-4eaa-422f-8de9-d6133a8b2016","Type":"ContainerDied","Data":"e16f8a87364f9124b8fa2ebe666b652656bef4fcb7b7b1b7a185fbeea9bd8939"} Oct 11 10:27:42.766915 master-1 kubenswrapper[4771]: I1011 10:27:42.766856 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/1.log" Oct 11 10:27:42.769133 master-1 kubenswrapper[4771]: I1011 10:27:42.767935 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" event={"ID":"e115f8be-9e65-4407-8111-568e5ea8ac1b","Type":"ContainerStarted","Data":"d6f0bf05ac57d47238297705efd6175b4b0b48e0ab73a222acdf287379d27829"} Oct 11 10:27:42.770532 master-1 kubenswrapper[4771]: I1011 10:27:42.770484 4771 generic.go:334] "Generic (PLEG): container finished" podID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerID="92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce" exitCode=0 Oct 11 10:27:42.770652 master-1 kubenswrapper[4771]: I1011 10:27:42.770621 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce"} Oct 11 10:27:42.774019 master-1 kubenswrapper[4771]: I1011 10:27:42.773962 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" event={"ID":"a65a56b0-5ee8-4429-8fe5-b33a6f29bc79","Type":"ContainerStarted","Data":"fd429044454b120b7f284bbe76d575ec34a3889f8f0e590f4edb09bf076942ee"} Oct 11 10:27:42.775421 master-1 kubenswrapper[4771]: I1011 10:27:42.775326 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-node-identity/network-node-identity-sk5cm" podStartSLOduration=1.029117293 podStartE2EDuration="11.775306544s" podCreationTimestamp="2025-10-11 10:27:31 +0000 UTC" firstStartedPulling="2025-10-11 10:27:31.543288911 +0000 UTC m=+83.517515352" lastFinishedPulling="2025-10-11 10:27:42.289478122 +0000 UTC m=+94.263704603" observedRunningTime="2025-10-11 10:27:42.775305684 +0000 UTC m=+94.749532195" watchObservedRunningTime="2025-10-11 10:27:42.775306544 +0000 UTC m=+94.749533045" Oct 11 10:27:42.795347 master-1 kubenswrapper[4771]: I1011 10:27:42.795273 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" podStartSLOduration=24.795252733 podStartE2EDuration="24.795252733s" podCreationTimestamp="2025-10-11 10:27:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:27:42.794999546 +0000 UTC m=+94.769226057" watchObservedRunningTime="2025-10-11 10:27:42.795252733 +0000 UTC m=+94.769479234" Oct 11 10:27:42.878822 master-1 kubenswrapper[4771]: I1011 10:27:42.878727 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-control-plane-864d695c77-5mflb" podStartSLOduration=1.6528184700000002 podStartE2EDuration="17.878698077s" podCreationTimestamp="2025-10-11 10:27:25 +0000 UTC" firstStartedPulling="2025-10-11 10:27:26.002937135 +0000 UTC m=+77.977163616" lastFinishedPulling="2025-10-11 10:27:42.228816742 +0000 UTC m=+94.203043223" observedRunningTime="2025-10-11 10:27:42.832908722 +0000 UTC m=+94.807135223" watchObservedRunningTime="2025-10-11 10:27:42.878698077 +0000 UTC m=+94.852924568" Oct 11 10:27:43.436555 master-1 kubenswrapper[4771]: I1011 10:27:43.436064 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:43.437964 master-1 kubenswrapper[4771]: E1011 10:27:43.436689 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:43.783505 master-1 kubenswrapper[4771]: I1011 10:27:43.783409 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b4dff81-4eaa-422f-8de9-d6133a8b2016" containerID="9bbb4a99213e3882e40e0fb093245b500c5a29883ced07500ad74a177d298364" exitCode=0 Oct 11 10:27:43.783698 master-1 kubenswrapper[4771]: I1011 10:27:43.783543 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" event={"ID":"0b4dff81-4eaa-422f-8de9-d6133a8b2016","Type":"ContainerDied","Data":"9bbb4a99213e3882e40e0fb093245b500c5a29883ced07500ad74a177d298364"} Oct 11 10:27:43.786540 master-1 kubenswrapper[4771]: I1011 10:27:43.786494 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/2.log" Oct 11 10:27:43.787598 master-1 kubenswrapper[4771]: I1011 10:27:43.787530 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/1.log" Oct 11 10:27:43.788611 master-1 kubenswrapper[4771]: I1011 10:27:43.788568 4771 generic.go:334] "Generic (PLEG): container finished" podID="e115f8be-9e65-4407-8111-568e5ea8ac1b" containerID="d6f0bf05ac57d47238297705efd6175b4b0b48e0ab73a222acdf287379d27829" exitCode=1 Oct 11 10:27:43.788666 master-1 kubenswrapper[4771]: I1011 10:27:43.788632 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" event={"ID":"e115f8be-9e65-4407-8111-568e5ea8ac1b","Type":"ContainerDied","Data":"d6f0bf05ac57d47238297705efd6175b4b0b48e0ab73a222acdf287379d27829"} Oct 11 10:27:43.788724 master-1 kubenswrapper[4771]: I1011 10:27:43.788702 4771 scope.go:117] "RemoveContainer" containerID="60a93fb0643f4c652f352528fa2f68ed301ee9978f9f9a0561174149a6bb77e4" Oct 11 10:27:43.790216 master-1 kubenswrapper[4771]: I1011 10:27:43.790164 4771 scope.go:117] "RemoveContainer" containerID="d6f0bf05ac57d47238297705efd6175b4b0b48e0ab73a222acdf287379d27829" Oct 11 10:27:43.790601 master-1 kubenswrapper[4771]: E1011 10:27:43.790495 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-779749f859-5xxzp_openshift-cloud-controller-manager-operator(e115f8be-9e65-4407-8111-568e5ea8ac1b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" podUID="e115f8be-9e65-4407-8111-568e5ea8ac1b" Oct 11 10:27:43.795099 master-1 kubenswrapper[4771]: I1011 10:27:43.795056 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerStarted","Data":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} Oct 11 10:27:43.795170 master-1 kubenswrapper[4771]: I1011 10:27:43.795103 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerStarted","Data":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} Oct 11 10:27:43.795170 master-1 kubenswrapper[4771]: I1011 10:27:43.795127 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerStarted","Data":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} Oct 11 10:27:43.795170 master-1 kubenswrapper[4771]: I1011 10:27:43.795145 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerStarted","Data":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} Oct 11 10:27:43.795170 master-1 kubenswrapper[4771]: I1011 10:27:43.795165 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerStarted","Data":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} Oct 11 10:27:43.795390 master-1 kubenswrapper[4771]: I1011 10:27:43.795182 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerStarted","Data":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} Oct 11 10:27:44.436884 master-1 kubenswrapper[4771]: I1011 10:27:44.436594 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:44.436884 master-1 kubenswrapper[4771]: E1011 10:27:44.436751 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:44.506672 master-1 kubenswrapper[4771]: I1011 10:27:44.506558 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:44.506952 master-1 kubenswrapper[4771]: E1011 10:27:44.506752 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 11 10:27:44.506952 master-1 kubenswrapper[4771]: E1011 10:27:44.506785 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 11 10:27:44.506952 master-1 kubenswrapper[4771]: E1011 10:27:44.506807 4771 projected.go:194] Error preparing data for projected volume kube-api-access-hktrh for pod openshift-network-diagnostics/network-check-target-4pm7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:44.506952 master-1 kubenswrapper[4771]: E1011 10:27:44.506901 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh podName:0bde275d-f0a5-4bea-93f7-edd2077e46b4 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:00.506868585 +0000 UTC m=+112.481095066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hktrh" (UniqueName: "kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh") pod "network-check-target-4pm7x" (UID: "0bde275d-f0a5-4bea-93f7-edd2077e46b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:27:44.799102 master-1 kubenswrapper[4771]: I1011 10:27:44.798919 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/2.log" Oct 11 10:27:44.803956 master-1 kubenswrapper[4771]: I1011 10:27:44.803891 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" event={"ID":"0b4dff81-4eaa-422f-8de9-d6133a8b2016","Type":"ContainerStarted","Data":"9c7f4a451768bb810cb6fa39c38e6c02760cfe7f21dbe9bd7c2ba963f6a4e616"} Oct 11 10:27:45.436530 master-1 kubenswrapper[4771]: I1011 10:27:45.436378 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:45.436831 master-1 kubenswrapper[4771]: E1011 10:27:45.436552 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:45.812656 master-1 kubenswrapper[4771]: I1011 10:27:45.812594 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerStarted","Data":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} Oct 11 10:27:45.919569 master-1 kubenswrapper[4771]: I1011 10:27:45.919478 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:45.919752 master-1 kubenswrapper[4771]: E1011 10:27:45.919676 4771 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:45.919845 master-1 kubenswrapper[4771]: E1011 10:27:45.919814 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs podName:2c084572-a5c9-4787-8a14-b7d6b0810a1b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:17.919778928 +0000 UTC m=+129.894005399 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs") pod "network-metrics-daemon-fgjvw" (UID: "2c084572-a5c9-4787-8a14-b7d6b0810a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 11 10:27:46.436020 master-1 kubenswrapper[4771]: I1011 10:27:46.435904 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:46.436423 master-1 kubenswrapper[4771]: E1011 10:27:46.436098 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:47.436779 master-1 kubenswrapper[4771]: I1011 10:27:47.436697 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:47.437479 master-1 kubenswrapper[4771]: E1011 10:27:47.437025 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:48.437091 master-1 kubenswrapper[4771]: I1011 10:27:48.436602 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:48.438321 master-1 kubenswrapper[4771]: E1011 10:27:48.437293 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:48.828466 master-1 kubenswrapper[4771]: I1011 10:27:48.828302 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerStarted","Data":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} Oct 11 10:27:48.828810 master-1 kubenswrapper[4771]: I1011 10:27:48.828764 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:48.828902 master-1 kubenswrapper[4771]: I1011 10:27:48.828828 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:48.828902 master-1 kubenswrapper[4771]: I1011 10:27:48.828852 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:48.864279 master-1 kubenswrapper[4771]: I1011 10:27:48.864187 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lvp6f" podStartSLOduration=8.128615693 podStartE2EDuration="35.864166913s" podCreationTimestamp="2025-10-11 10:27:13 +0000 UTC" firstStartedPulling="2025-10-11 10:27:14.433405404 +0000 UTC m=+66.407631845" lastFinishedPulling="2025-10-11 10:27:42.168956624 +0000 UTC m=+94.143183065" observedRunningTime="2025-10-11 10:27:44.827564135 +0000 UTC m=+96.801790596" watchObservedRunningTime="2025-10-11 10:27:48.864166913 +0000 UTC m=+100.838393394" Oct 11 10:27:49.436723 master-1 kubenswrapper[4771]: I1011 10:27:49.436530 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:49.436723 master-1 kubenswrapper[4771]: E1011 10:27:49.436736 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:50.403466 master-1 kubenswrapper[4771]: I1011 10:27:50.403374 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" podStartSLOduration=9.038853162 podStartE2EDuration="25.403324257s" podCreationTimestamp="2025-10-11 10:27:25 +0000 UTC" firstStartedPulling="2025-10-11 10:27:25.936260376 +0000 UTC m=+77.910486827" lastFinishedPulling="2025-10-11 10:27:42.300731481 +0000 UTC m=+94.274957922" observedRunningTime="2025-10-11 10:27:48.864030629 +0000 UTC m=+100.838257170" watchObservedRunningTime="2025-10-11 10:27:50.403324257 +0000 UTC m=+102.377550718" Oct 11 10:27:50.404422 master-1 kubenswrapper[4771]: I1011 10:27:50.404318 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4pm7x"] Oct 11 10:27:50.404596 master-1 kubenswrapper[4771]: I1011 10:27:50.404561 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:50.404799 master-1 kubenswrapper[4771]: E1011 10:27:50.404751 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:50.405295 master-1 kubenswrapper[4771]: I1011 10:27:50.405268 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fgjvw"] Oct 11 10:27:50.405563 master-1 kubenswrapper[4771]: I1011 10:27:50.405544 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:50.405780 master-1 kubenswrapper[4771]: E1011 10:27:50.405754 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:51.031675 master-1 kubenswrapper[4771]: I1011 10:27:51.031591 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-fl2bs"] Oct 11 10:27:51.034053 master-1 kubenswrapper[4771]: I1011 10:27:51.032025 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovn-controller" containerID="cri-o://2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88" gracePeriod=30 Oct 11 10:27:51.034053 master-1 kubenswrapper[4771]: I1011 10:27:51.032052 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="nbdb" containerID="cri-o://676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853" gracePeriod=30 Oct 11 10:27:51.034053 master-1 kubenswrapper[4771]: I1011 10:27:51.032098 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="northd" containerID="cri-o://7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3" gracePeriod=30 Oct 11 10:27:51.034053 master-1 kubenswrapper[4771]: I1011 10:27:51.032124 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kube-rbac-proxy-ovn-metrics" containerID="cri-o://ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe" gracePeriod=30 Oct 11 10:27:51.034053 master-1 kubenswrapper[4771]: I1011 10:27:51.032198 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="sbdb" containerID="cri-o://0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1" gracePeriod=30 Oct 11 10:27:51.034053 master-1 kubenswrapper[4771]: I1011 10:27:51.032338 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kube-rbac-proxy-node" containerID="cri-o://cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0" gracePeriod=30 Oct 11 10:27:51.034053 master-1 kubenswrapper[4771]: I1011 10:27:51.032338 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovn-acl-logging" containerID="cri-o://fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79" gracePeriod=30 Oct 11 10:27:51.095884 master-1 kubenswrapper[4771]: I1011 10:27:51.095782 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovnkube-controller" containerID="cri-o://2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc" gracePeriod=30 Oct 11 10:27:51.366673 master-1 kubenswrapper[4771]: I1011 10:27:51.366614 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/ovnkube-controller/0.log" Oct 11 10:27:51.369306 master-1 kubenswrapper[4771]: I1011 10:27:51.369264 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/kube-rbac-proxy-ovn-metrics/0.log" Oct 11 10:27:51.370408 master-1 kubenswrapper[4771]: I1011 10:27:51.370321 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/kube-rbac-proxy-node/0.log" Oct 11 10:27:51.371029 master-1 kubenswrapper[4771]: I1011 10:27:51.370984 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/ovn-acl-logging/0.log" Oct 11 10:27:51.371935 master-1 kubenswrapper[4771]: I1011 10:27:51.371891 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/ovn-controller/0.log" Oct 11 10:27:51.372540 master-1 kubenswrapper[4771]: I1011 10:27:51.372497 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:51.391454 master-1 kubenswrapper[4771]: I1011 10:27:51.391240 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-config\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.391454 master-1 kubenswrapper[4771]: I1011 10:27:51.391343 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-slash\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.391454 master-1 kubenswrapper[4771]: I1011 10:27:51.391422 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42mmb\" (UniqueName: \"kubernetes.io/projected/96c2d0f1-e436-480c-9e34-9068178f9df4-kube-api-access-42mmb\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.391454 master-1 kubenswrapper[4771]: I1011 10:27:51.391458 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-systemd\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.391735 master-1 kubenswrapper[4771]: I1011 10:27:51.391470 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-slash" (OuterVolumeSpecName: "host-slash") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "host-slash". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.391735 master-1 kubenswrapper[4771]: I1011 10:27:51.391492 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-env-overrides\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.391735 master-1 kubenswrapper[4771]: I1011 10:27:51.391523 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-ovn\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.391735 master-1 kubenswrapper[4771]: I1011 10:27:51.391566 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-bin\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.391735 master-1 kubenswrapper[4771]: I1011 10:27:51.391606 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-node-log\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.391879 master-1 kubenswrapper[4771]: I1011 10:27:51.391761 4771 reconciler_common.go:293] "Volume detached for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-slash\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.391879 master-1 kubenswrapper[4771]: I1011 10:27:51.391836 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-node-log" (OuterVolumeSpecName: "node-log") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "node-log". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.391949 master-1 kubenswrapper[4771]: I1011 10:27:51.391878 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-config" (OuterVolumeSpecName: "ovnkube-config") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "ovnkube-config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:27:51.391949 master-1 kubenswrapper[4771]: I1011 10:27:51.391905 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-ovn" (OuterVolumeSpecName: "run-ovn") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "run-ovn". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.391949 master-1 kubenswrapper[4771]: I1011 10:27:51.391923 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-bin" (OuterVolumeSpecName: "host-cni-bin") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "host-cni-bin". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.392377 master-1 kubenswrapper[4771]: I1011 10:27:51.392302 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-env-overrides" (OuterVolumeSpecName: "env-overrides") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "env-overrides". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:27:51.397403 master-1 kubenswrapper[4771]: I1011 10:27:51.397313 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c2d0f1-e436-480c-9e34-9068178f9df4-kube-api-access-42mmb" (OuterVolumeSpecName: "kube-api-access-42mmb") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "kube-api-access-42mmb". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:27:51.400841 master-1 kubenswrapper[4771]: I1011 10:27:51.400785 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-systemd" (OuterVolumeSpecName: "run-systemd") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "run-systemd". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427331 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-p9l4v"] Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: E1011 10:27:51.427511 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kubecfg-setup" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427533 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kubecfg-setup" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: E1011 10:27:51.427548 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="northd" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427561 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="northd" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: E1011 10:27:51.427575 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="nbdb" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427588 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="nbdb" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: E1011 10:27:51.427602 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovn-acl-logging" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427614 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovn-acl-logging" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: E1011 10:27:51.427626 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="sbdb" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427638 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="sbdb" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: E1011 10:27:51.427651 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovnkube-controller" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427664 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovnkube-controller" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: E1011 10:27:51.427677 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kube-rbac-proxy-node" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427688 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kube-rbac-proxy-node" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: E1011 10:27:51.427701 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kube-rbac-proxy-ovn-metrics" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427713 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kube-rbac-proxy-ovn-metrics" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: E1011 10:27:51.427727 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovn-controller" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427739 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovn-controller" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427795 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovn-controller" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427814 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kube-rbac-proxy-ovn-metrics" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427831 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="nbdb" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427849 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovnkube-controller" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427863 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="kube-rbac-proxy-node" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427878 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="ovn-acl-logging" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427892 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="northd" Oct 11 10:27:51.428509 master-1 kubenswrapper[4771]: I1011 10:27:51.427907 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerName="sbdb" Oct 11 10:27:51.435587 master-1 kubenswrapper[4771]: I1011 10:27:51.435501 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.492394 master-1 kubenswrapper[4771]: I1011 10:27:51.492206 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-script-lib\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.492766 master-1 kubenswrapper[4771]: I1011 10:27:51.492487 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-openvswitch\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.492766 master-1 kubenswrapper[4771]: I1011 10:27:51.492611 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-openvswitch" (OuterVolumeSpecName: "run-openvswitch") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "run-openvswitch". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.492845 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-systemd-units\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.492896 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.492930 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-netns\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.492961 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-ovn-kubernetes\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493001 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96c2d0f1-e436-480c-9e34-9068178f9df4-ovn-node-metrics-cert\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493039 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-var-lib-openvswitch\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.492975 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-systemd-units" (OuterVolumeSpecName: "systemd-units") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "systemd-units". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493043 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-netns" (OuterVolumeSpecName: "host-run-netns") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "host-run-netns". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493079 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-log-socket\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493036 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-ovn-kubernetes" (OuterVolumeSpecName: "host-run-ovn-kubernetes") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "host-run-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493105 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-var-lib-openvswitch" (OuterVolumeSpecName: "var-lib-openvswitch") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "var-lib-openvswitch". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493122 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-kubelet\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493167 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-netd\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493172 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-script-lib" (OuterVolumeSpecName: "ovnkube-script-lib") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "ovnkube-script-lib". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493194 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-log-socket" (OuterVolumeSpecName: "log-socket") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "log-socket". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493232 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-netd" (OuterVolumeSpecName: "host-cni-netd") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "host-cni-netd". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.493348 master-1 kubenswrapper[4771]: I1011 10:27:51.493210 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-etc-openvswitch\") pod \"96c2d0f1-e436-480c-9e34-9068178f9df4\" (UID: \"96c2d0f1-e436-480c-9e34-9068178f9df4\") " Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493163 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-var-lib-cni-networks-ovn-kubernetes" (OuterVolumeSpecName: "host-var-lib-cni-networks-ovn-kubernetes") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "host-var-lib-cni-networks-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493260 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-etc-openvswitch" (OuterVolumeSpecName: "etc-openvswitch") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "etc-openvswitch". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493239 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-kubelet" (OuterVolumeSpecName: "host-kubelet") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "host-kubelet". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493424 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-run-systemd\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493500 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm62v\" (UniqueName: \"kubernetes.io/projected/a199ebda-03a4-4154-902b-28397e4bc616-kube-api-access-tm62v\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493551 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-run-ovn\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493588 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-slash\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493620 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-log-socket\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493653 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-systemd-units\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493688 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493819 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-run-openvswitch\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.493953 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-var-lib-openvswitch\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.494055 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a199ebda-03a4-4154-902b-28397e4bc616-env-overrides\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.494109 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-kubelet\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.494155 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-cni-bin\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.494322 master-1 kubenswrapper[4771]: I1011 10:27:51.494200 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a199ebda-03a4-4154-902b-28397e4bc616-ovnkube-config\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494243 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-node-log\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494283 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-run-netns\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494329 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a199ebda-03a4-4154-902b-28397e4bc616-ovnkube-script-lib\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494410 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-run-ovn-kubernetes\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494457 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a199ebda-03a4-4154-902b-28397e4bc616-ovn-node-metrics-cert\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494529 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-etc-openvswitch\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494571 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-cni-netd\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494667 4771 reconciler_common.go:293] "Volume detached for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-env-overrides\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494697 4771 reconciler_common.go:293] "Volume detached for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-ovn\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494716 4771 reconciler_common.go:293] "Volume detached for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-bin\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494736 4771 reconciler_common.go:293] "Volume detached for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-node-log\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494755 4771 reconciler_common.go:293] "Volume detached for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-var-lib-openvswitch\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494771 4771 reconciler_common.go:293] "Volume detached for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-log-socket\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494789 4771 reconciler_common.go:293] "Volume detached for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-cni-netd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494806 4771 reconciler_common.go:293] "Volume detached for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-kubelet\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494823 4771 reconciler_common.go:293] "Volume detached for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-etc-openvswitch\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494840 4771 reconciler_common.go:293] "Volume detached for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-script-lib\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494857 4771 reconciler_common.go:293] "Volume detached for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-openvswitch\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494874 4771 reconciler_common.go:293] "Volume detached for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-systemd-units\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494891 4771 reconciler_common.go:293] "Volume detached for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-netns\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494908 4771 reconciler_common.go:293] "Volume detached for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-run-ovn-kubernetes\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.495170 master-1 kubenswrapper[4771]: I1011 10:27:51.494926 4771 reconciler_common.go:293] "Volume detached for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-host-var-lib-cni-networks-ovn-kubernetes\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.496467 master-1 kubenswrapper[4771]: I1011 10:27:51.494945 4771 reconciler_common.go:293] "Volume detached for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96c2d0f1-e436-480c-9e34-9068178f9df4-ovnkube-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.496467 master-1 kubenswrapper[4771]: I1011 10:27:51.494962 4771 reconciler_common.go:293] "Volume detached for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96c2d0f1-e436-480c-9e34-9068178f9df4-run-systemd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.496467 master-1 kubenswrapper[4771]: I1011 10:27:51.494979 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-42mmb\" (UniqueName: \"kubernetes.io/projected/96c2d0f1-e436-480c-9e34-9068178f9df4-kube-api-access-42mmb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.498091 master-1 kubenswrapper[4771]: I1011 10:27:51.498025 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c2d0f1-e436-480c-9e34-9068178f9df4-ovn-node-metrics-cert" (OuterVolumeSpecName: "ovn-node-metrics-cert") pod "96c2d0f1-e436-480c-9e34-9068178f9df4" (UID: "96c2d0f1-e436-480c-9e34-9068178f9df4"). InnerVolumeSpecName "ovn-node-metrics-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:27:51.595584 master-1 kubenswrapper[4771]: I1011 10:27:51.595404 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-var-lib-openvswitch\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.595584 master-1 kubenswrapper[4771]: I1011 10:27:51.595460 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a199ebda-03a4-4154-902b-28397e4bc616-env-overrides\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.595584 master-1 kubenswrapper[4771]: I1011 10:27:51.595494 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-kubelet\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.595584 master-1 kubenswrapper[4771]: I1011 10:27:51.595528 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-node-log\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.595584 master-1 kubenswrapper[4771]: I1011 10:27:51.595559 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-cni-bin\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.595584 master-1 kubenswrapper[4771]: I1011 10:27:51.595573 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-var-lib-openvswitch\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595595 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a199ebda-03a4-4154-902b-28397e4bc616-ovnkube-config\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595707 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-run-netns\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595733 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-kubelet\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595756 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a199ebda-03a4-4154-902b-28397e4bc616-ovnkube-script-lib\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595751 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-node-log\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595807 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-run-netns\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595810 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-run-ovn-kubernetes\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595862 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a199ebda-03a4-4154-902b-28397e4bc616-ovn-node-metrics-cert\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595880 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-cni-bin\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595914 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-etc-openvswitch\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595945 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-cni-netd\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.595978 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-run-systemd\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.596010 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-tm62v\" (UniqueName: \"kubernetes.io/projected/a199ebda-03a4-4154-902b-28397e4bc616-kube-api-access-tm62v\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.596041 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-slash\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.596072 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-run-ovn\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.596105 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-log-socket\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.596520 master-1 kubenswrapper[4771]: I1011 10:27:51.596118 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-etc-openvswitch\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596140 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-systemd-units\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596163 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-run-systemd\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596181 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-systemd-units\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.595863 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-run-ovn-kubernetes\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596261 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-cni-netd\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596337 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-log-socket\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596349 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-slash\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596344 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-run-ovn\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596724 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a199ebda-03a4-4154-902b-28397e4bc616-env-overrides\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596805 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596858 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-run-openvswitch\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596910 4771 reconciler_common.go:293] "Volume detached for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96c2d0f1-e436-480c-9e34-9068178f9df4-ovn-node-metrics-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596930 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596958 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a199ebda-03a4-4154-902b-28397e4bc616-run-openvswitch\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.596935 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a199ebda-03a4-4154-902b-28397e4bc616-ovnkube-config\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.597530 master-1 kubenswrapper[4771]: I1011 10:27:51.597288 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a199ebda-03a4-4154-902b-28397e4bc616-ovnkube-script-lib\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.600891 master-1 kubenswrapper[4771]: I1011 10:27:51.600839 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a199ebda-03a4-4154-902b-28397e4bc616-ovn-node-metrics-cert\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.623303 master-1 kubenswrapper[4771]: I1011 10:27:51.623200 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm62v\" (UniqueName: \"kubernetes.io/projected/a199ebda-03a4-4154-902b-28397e4bc616-kube-api-access-tm62v\") pod \"ovnkube-node-p9l4v\" (UID: \"a199ebda-03a4-4154-902b-28397e4bc616\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.756428 master-1 kubenswrapper[4771]: I1011 10:27:51.756334 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:51.776365 master-1 kubenswrapper[4771]: W1011 10:27:51.776285 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda199ebda_03a4_4154_902b_28397e4bc616.slice/crio-c4fa7112a50f60bb3eac55810697b99dd94789b499b8f14995a67cee7dbf8ed0 WatchSource:0}: Error finding container c4fa7112a50f60bb3eac55810697b99dd94789b499b8f14995a67cee7dbf8ed0: Status 404 returned error can't find the container with id c4fa7112a50f60bb3eac55810697b99dd94789b499b8f14995a67cee7dbf8ed0 Oct 11 10:27:51.841896 master-1 kubenswrapper[4771]: I1011 10:27:51.841445 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/ovnkube-controller/0.log" Oct 11 10:27:51.844989 master-1 kubenswrapper[4771]: I1011 10:27:51.844924 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/kube-rbac-proxy-ovn-metrics/0.log" Oct 11 10:27:51.845929 master-1 kubenswrapper[4771]: I1011 10:27:51.845812 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/kube-rbac-proxy-node/0.log" Oct 11 10:27:51.846577 master-1 kubenswrapper[4771]: I1011 10:27:51.846532 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/ovn-acl-logging/0.log" Oct 11 10:27:51.847350 master-1 kubenswrapper[4771]: I1011 10:27:51.847294 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl2bs_96c2d0f1-e436-480c-9e34-9068178f9df4/ovn-controller/0.log" Oct 11 10:27:51.848134 master-1 kubenswrapper[4771]: I1011 10:27:51.848064 4771 generic.go:334] "Generic (PLEG): container finished" podID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerID="2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc" exitCode=1 Oct 11 10:27:51.848134 master-1 kubenswrapper[4771]: I1011 10:27:51.848121 4771 generic.go:334] "Generic (PLEG): container finished" podID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerID="0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1" exitCode=0 Oct 11 10:27:51.848134 master-1 kubenswrapper[4771]: I1011 10:27:51.848138 4771 generic.go:334] "Generic (PLEG): container finished" podID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerID="676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853" exitCode=0 Oct 11 10:27:51.848472 master-1 kubenswrapper[4771]: I1011 10:27:51.848140 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} Oct 11 10:27:51.848472 master-1 kubenswrapper[4771]: I1011 10:27:51.848157 4771 generic.go:334] "Generic (PLEG): container finished" podID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerID="7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3" exitCode=0 Oct 11 10:27:51.848472 master-1 kubenswrapper[4771]: I1011 10:27:51.848210 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} Oct 11 10:27:51.848472 master-1 kubenswrapper[4771]: I1011 10:27:51.848230 4771 generic.go:334] "Generic (PLEG): container finished" podID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerID="ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe" exitCode=143 Oct 11 10:27:51.848472 master-1 kubenswrapper[4771]: I1011 10:27:51.848236 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} Oct 11 10:27:51.848472 master-1 kubenswrapper[4771]: I1011 10:27:51.848262 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} Oct 11 10:27:51.848472 master-1 kubenswrapper[4771]: I1011 10:27:51.848250 4771 generic.go:334] "Generic (PLEG): container finished" podID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerID="cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0" exitCode=143 Oct 11 10:27:51.848472 master-1 kubenswrapper[4771]: I1011 10:27:51.848301 4771 scope.go:117] "RemoveContainer" containerID="2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc" Oct 11 10:27:51.848472 master-1 kubenswrapper[4771]: I1011 10:27:51.848302 4771 generic.go:334] "Generic (PLEG): container finished" podID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerID="fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79" exitCode=143 Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848503 4771 generic.go:334] "Generic (PLEG): container finished" podID="96c2d0f1-e436-480c-9e34-9068178f9df4" containerID="2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88" exitCode=143 Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848282 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848612 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848636 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848655 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848668 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848696 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848713 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848725 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848737 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848205 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848748 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848794 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848806 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848847 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848860 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848871 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848943 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848968 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848981 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.848992 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.849038 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.849052 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.849062 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.849072 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.849083 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} Oct 11 10:27:51.849092 master-1 kubenswrapper[4771]: I1011 10:27:51.849094 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849111 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl2bs" event={"ID":"96c2d0f1-e436-480c-9e34-9068178f9df4","Type":"ContainerDied","Data":"e331eefac7c84cea9d904666f2942a1490b780d5057ea46b8fe92374c1ddb75a"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849129 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849144 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849157 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849171 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849188 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849200 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849212 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849223 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.849235 4771 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce"} Oct 11 10:27:51.850822 master-1 kubenswrapper[4771]: I1011 10:27:51.850435 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerStarted","Data":"c4fa7112a50f60bb3eac55810697b99dd94789b499b8f14995a67cee7dbf8ed0"} Oct 11 10:27:51.870552 master-1 kubenswrapper[4771]: I1011 10:27:51.870494 4771 scope.go:117] "RemoveContainer" containerID="0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1" Oct 11 10:27:51.888563 master-1 kubenswrapper[4771]: I1011 10:27:51.888512 4771 scope.go:117] "RemoveContainer" containerID="676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853" Oct 11 10:27:51.893719 master-1 kubenswrapper[4771]: I1011 10:27:51.893670 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-fl2bs"] Oct 11 10:27:51.898075 master-1 kubenswrapper[4771]: I1011 10:27:51.898002 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-fl2bs"] Oct 11 10:27:51.906820 master-1 kubenswrapper[4771]: I1011 10:27:51.906761 4771 scope.go:117] "RemoveContainer" containerID="7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3" Oct 11 10:27:51.967967 master-1 kubenswrapper[4771]: I1011 10:27:51.967913 4771 scope.go:117] "RemoveContainer" containerID="ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe" Oct 11 10:27:51.984039 master-1 kubenswrapper[4771]: I1011 10:27:51.983995 4771 scope.go:117] "RemoveContainer" containerID="cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0" Oct 11 10:27:51.998677 master-1 kubenswrapper[4771]: I1011 10:27:51.998630 4771 scope.go:117] "RemoveContainer" containerID="fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79" Oct 11 10:27:52.012420 master-1 kubenswrapper[4771]: I1011 10:27:52.012312 4771 scope.go:117] "RemoveContainer" containerID="2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88" Oct 11 10:27:52.029065 master-1 kubenswrapper[4771]: I1011 10:27:52.029006 4771 scope.go:117] "RemoveContainer" containerID="92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce" Oct 11 10:27:52.045892 master-1 kubenswrapper[4771]: I1011 10:27:52.045497 4771 scope.go:117] "RemoveContainer" containerID="2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc" Oct 11 10:27:52.046946 master-1 kubenswrapper[4771]: E1011 10:27:52.045949 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": container with ID starting with 2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc not found: ID does not exist" containerID="2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc" Oct 11 10:27:52.046946 master-1 kubenswrapper[4771]: I1011 10:27:52.045996 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} err="failed to get container status \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": rpc error: code = NotFound desc = could not find container \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": container with ID starting with 2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc not found: ID does not exist" Oct 11 10:27:52.046946 master-1 kubenswrapper[4771]: I1011 10:27:52.046035 4771 scope.go:117] "RemoveContainer" containerID="0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1" Oct 11 10:27:52.046946 master-1 kubenswrapper[4771]: E1011 10:27:52.046621 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": container with ID starting with 0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1 not found: ID does not exist" containerID="0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1" Oct 11 10:27:52.046946 master-1 kubenswrapper[4771]: I1011 10:27:52.046675 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} err="failed to get container status \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": rpc error: code = NotFound desc = could not find container \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": container with ID starting with 0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1 not found: ID does not exist" Oct 11 10:27:52.046946 master-1 kubenswrapper[4771]: I1011 10:27:52.046714 4771 scope.go:117] "RemoveContainer" containerID="676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853" Oct 11 10:27:52.047296 master-1 kubenswrapper[4771]: E1011 10:27:52.047103 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": container with ID starting with 676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853 not found: ID does not exist" containerID="676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853" Oct 11 10:27:52.047296 master-1 kubenswrapper[4771]: I1011 10:27:52.047140 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} err="failed to get container status \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": rpc error: code = NotFound desc = could not find container \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": container with ID starting with 676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853 not found: ID does not exist" Oct 11 10:27:52.047296 master-1 kubenswrapper[4771]: I1011 10:27:52.047169 4771 scope.go:117] "RemoveContainer" containerID="7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3" Oct 11 10:27:52.048021 master-1 kubenswrapper[4771]: E1011 10:27:52.047961 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": container with ID starting with 7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3 not found: ID does not exist" containerID="7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3" Oct 11 10:27:52.048087 master-1 kubenswrapper[4771]: I1011 10:27:52.048010 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} err="failed to get container status \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": rpc error: code = NotFound desc = could not find container \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": container with ID starting with 7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3 not found: ID does not exist" Oct 11 10:27:52.048087 master-1 kubenswrapper[4771]: I1011 10:27:52.048040 4771 scope.go:117] "RemoveContainer" containerID="ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe" Oct 11 10:27:52.048582 master-1 kubenswrapper[4771]: E1011 10:27:52.048531 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": container with ID starting with ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe not found: ID does not exist" containerID="ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe" Oct 11 10:27:52.048667 master-1 kubenswrapper[4771]: I1011 10:27:52.048573 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} err="failed to get container status \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": rpc error: code = NotFound desc = could not find container \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": container with ID starting with ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe not found: ID does not exist" Oct 11 10:27:52.048667 master-1 kubenswrapper[4771]: I1011 10:27:52.048599 4771 scope.go:117] "RemoveContainer" containerID="cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0" Oct 11 10:27:52.049018 master-1 kubenswrapper[4771]: E1011 10:27:52.048970 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": container with ID starting with cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0 not found: ID does not exist" containerID="cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0" Oct 11 10:27:52.049113 master-1 kubenswrapper[4771]: I1011 10:27:52.049009 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} err="failed to get container status \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": rpc error: code = NotFound desc = could not find container \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": container with ID starting with cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0 not found: ID does not exist" Oct 11 10:27:52.049113 master-1 kubenswrapper[4771]: I1011 10:27:52.049035 4771 scope.go:117] "RemoveContainer" containerID="fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79" Oct 11 10:27:52.049590 master-1 kubenswrapper[4771]: E1011 10:27:52.049542 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79\": container with ID starting with fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79 not found: ID does not exist" containerID="fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79" Oct 11 10:27:52.049658 master-1 kubenswrapper[4771]: I1011 10:27:52.049581 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} err="failed to get container status \"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79\": rpc error: code = NotFound desc = could not find container \"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79\": container with ID starting with fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79 not found: ID does not exist" Oct 11 10:27:52.049658 master-1 kubenswrapper[4771]: I1011 10:27:52.049610 4771 scope.go:117] "RemoveContainer" containerID="2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88" Oct 11 10:27:52.050069 master-1 kubenswrapper[4771]: E1011 10:27:52.050014 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88\": container with ID starting with 2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88 not found: ID does not exist" containerID="2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88" Oct 11 10:27:52.050140 master-1 kubenswrapper[4771]: I1011 10:27:52.050064 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} err="failed to get container status \"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88\": rpc error: code = NotFound desc = could not find container \"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88\": container with ID starting with 2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88 not found: ID does not exist" Oct 11 10:27:52.050140 master-1 kubenswrapper[4771]: I1011 10:27:52.050093 4771 scope.go:117] "RemoveContainer" containerID="92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce" Oct 11 10:27:52.050634 master-1 kubenswrapper[4771]: E1011 10:27:52.050556 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce\": container with ID starting with 92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce not found: ID does not exist" containerID="92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce" Oct 11 10:27:52.050722 master-1 kubenswrapper[4771]: I1011 10:27:52.050651 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce"} err="failed to get container status \"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce\": rpc error: code = NotFound desc = could not find container \"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce\": container with ID starting with 92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce not found: ID does not exist" Oct 11 10:27:52.050784 master-1 kubenswrapper[4771]: I1011 10:27:52.050730 4771 scope.go:117] "RemoveContainer" containerID="2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc" Oct 11 10:27:52.051296 master-1 kubenswrapper[4771]: I1011 10:27:52.051232 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} err="failed to get container status \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": rpc error: code = NotFound desc = could not find container \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": container with ID starting with 2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc not found: ID does not exist" Oct 11 10:27:52.051296 master-1 kubenswrapper[4771]: I1011 10:27:52.051270 4771 scope.go:117] "RemoveContainer" containerID="0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1" Oct 11 10:27:52.051774 master-1 kubenswrapper[4771]: I1011 10:27:52.051707 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} err="failed to get container status \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": rpc error: code = NotFound desc = could not find container \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": container with ID starting with 0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1 not found: ID does not exist" Oct 11 10:27:52.051774 master-1 kubenswrapper[4771]: I1011 10:27:52.051763 4771 scope.go:117] "RemoveContainer" containerID="676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853" Oct 11 10:27:52.052230 master-1 kubenswrapper[4771]: I1011 10:27:52.052177 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} err="failed to get container status \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": rpc error: code = NotFound desc = could not find container \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": container with ID starting with 676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853 not found: ID does not exist" Oct 11 10:27:52.052230 master-1 kubenswrapper[4771]: I1011 10:27:52.052220 4771 scope.go:117] "RemoveContainer" containerID="7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3" Oct 11 10:27:52.052643 master-1 kubenswrapper[4771]: I1011 10:27:52.052582 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} err="failed to get container status \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": rpc error: code = NotFound desc = could not find container \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": container with ID starting with 7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3 not found: ID does not exist" Oct 11 10:27:52.052643 master-1 kubenswrapper[4771]: I1011 10:27:52.052624 4771 scope.go:117] "RemoveContainer" containerID="ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe" Oct 11 10:27:52.053155 master-1 kubenswrapper[4771]: I1011 10:27:52.053054 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} err="failed to get container status \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": rpc error: code = NotFound desc = could not find container \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": container with ID starting with ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe not found: ID does not exist" Oct 11 10:27:52.053155 master-1 kubenswrapper[4771]: I1011 10:27:52.053129 4771 scope.go:117] "RemoveContainer" containerID="cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0" Oct 11 10:27:52.053585 master-1 kubenswrapper[4771]: I1011 10:27:52.053529 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} err="failed to get container status \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": rpc error: code = NotFound desc = could not find container \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": container with ID starting with cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0 not found: ID does not exist" Oct 11 10:27:52.053585 master-1 kubenswrapper[4771]: I1011 10:27:52.053572 4771 scope.go:117] "RemoveContainer" containerID="fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79" Oct 11 10:27:52.054073 master-1 kubenswrapper[4771]: I1011 10:27:52.054002 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} err="failed to get container status \"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79\": rpc error: code = NotFound desc = could not find container \"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79\": container with ID starting with fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79 not found: ID does not exist" Oct 11 10:27:52.054073 master-1 kubenswrapper[4771]: I1011 10:27:52.054060 4771 scope.go:117] "RemoveContainer" containerID="2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88" Oct 11 10:27:52.054630 master-1 kubenswrapper[4771]: I1011 10:27:52.054574 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} err="failed to get container status \"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88\": rpc error: code = NotFound desc = could not find container \"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88\": container with ID starting with 2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88 not found: ID does not exist" Oct 11 10:27:52.054630 master-1 kubenswrapper[4771]: I1011 10:27:52.054615 4771 scope.go:117] "RemoveContainer" containerID="92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce" Oct 11 10:27:52.055070 master-1 kubenswrapper[4771]: I1011 10:27:52.055015 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce"} err="failed to get container status \"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce\": rpc error: code = NotFound desc = could not find container \"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce\": container with ID starting with 92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce not found: ID does not exist" Oct 11 10:27:52.055070 master-1 kubenswrapper[4771]: I1011 10:27:52.055054 4771 scope.go:117] "RemoveContainer" containerID="2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc" Oct 11 10:27:52.055474 master-1 kubenswrapper[4771]: I1011 10:27:52.055417 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} err="failed to get container status \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": rpc error: code = NotFound desc = could not find container \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": container with ID starting with 2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc not found: ID does not exist" Oct 11 10:27:52.055474 master-1 kubenswrapper[4771]: I1011 10:27:52.055458 4771 scope.go:117] "RemoveContainer" containerID="0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1" Oct 11 10:27:52.055948 master-1 kubenswrapper[4771]: I1011 10:27:52.055895 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} err="failed to get container status \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": rpc error: code = NotFound desc = could not find container \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": container with ID starting with 0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1 not found: ID does not exist" Oct 11 10:27:52.055948 master-1 kubenswrapper[4771]: I1011 10:27:52.055935 4771 scope.go:117] "RemoveContainer" containerID="676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853" Oct 11 10:27:52.056379 master-1 kubenswrapper[4771]: I1011 10:27:52.056306 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} err="failed to get container status \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": rpc error: code = NotFound desc = could not find container \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": container with ID starting with 676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853 not found: ID does not exist" Oct 11 10:27:52.056379 master-1 kubenswrapper[4771]: I1011 10:27:52.056345 4771 scope.go:117] "RemoveContainer" containerID="7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3" Oct 11 10:27:52.056836 master-1 kubenswrapper[4771]: I1011 10:27:52.056785 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} err="failed to get container status \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": rpc error: code = NotFound desc = could not find container \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": container with ID starting with 7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3 not found: ID does not exist" Oct 11 10:27:52.056836 master-1 kubenswrapper[4771]: I1011 10:27:52.056821 4771 scope.go:117] "RemoveContainer" containerID="ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe" Oct 11 10:27:52.057267 master-1 kubenswrapper[4771]: I1011 10:27:52.057211 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} err="failed to get container status \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": rpc error: code = NotFound desc = could not find container \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": container with ID starting with ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe not found: ID does not exist" Oct 11 10:27:52.057267 master-1 kubenswrapper[4771]: I1011 10:27:52.057254 4771 scope.go:117] "RemoveContainer" containerID="cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0" Oct 11 10:27:52.057689 master-1 kubenswrapper[4771]: I1011 10:27:52.057639 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} err="failed to get container status \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": rpc error: code = NotFound desc = could not find container \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": container with ID starting with cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0 not found: ID does not exist" Oct 11 10:27:52.057689 master-1 kubenswrapper[4771]: I1011 10:27:52.057674 4771 scope.go:117] "RemoveContainer" containerID="fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79" Oct 11 10:27:52.058109 master-1 kubenswrapper[4771]: I1011 10:27:52.058059 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} err="failed to get container status \"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79\": rpc error: code = NotFound desc = could not find container \"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79\": container with ID starting with fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79 not found: ID does not exist" Oct 11 10:27:52.058109 master-1 kubenswrapper[4771]: I1011 10:27:52.058094 4771 scope.go:117] "RemoveContainer" containerID="2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88" Oct 11 10:27:52.058513 master-1 kubenswrapper[4771]: I1011 10:27:52.058464 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} err="failed to get container status \"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88\": rpc error: code = NotFound desc = could not find container \"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88\": container with ID starting with 2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88 not found: ID does not exist" Oct 11 10:27:52.058513 master-1 kubenswrapper[4771]: I1011 10:27:52.058500 4771 scope.go:117] "RemoveContainer" containerID="92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce" Oct 11 10:27:52.058956 master-1 kubenswrapper[4771]: I1011 10:27:52.058905 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce"} err="failed to get container status \"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce\": rpc error: code = NotFound desc = could not find container \"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce\": container with ID starting with 92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce not found: ID does not exist" Oct 11 10:27:52.058956 master-1 kubenswrapper[4771]: I1011 10:27:52.058941 4771 scope.go:117] "RemoveContainer" containerID="2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc" Oct 11 10:27:52.060241 master-1 kubenswrapper[4771]: I1011 10:27:52.060173 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} err="failed to get container status \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": rpc error: code = NotFound desc = could not find container \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": container with ID starting with 2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc not found: ID does not exist" Oct 11 10:27:52.060241 master-1 kubenswrapper[4771]: I1011 10:27:52.060233 4771 scope.go:117] "RemoveContainer" containerID="0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1" Oct 11 10:27:52.060709 master-1 kubenswrapper[4771]: I1011 10:27:52.060660 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} err="failed to get container status \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": rpc error: code = NotFound desc = could not find container \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": container with ID starting with 0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1 not found: ID does not exist" Oct 11 10:27:52.060709 master-1 kubenswrapper[4771]: I1011 10:27:52.060696 4771 scope.go:117] "RemoveContainer" containerID="676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853" Oct 11 10:27:52.061226 master-1 kubenswrapper[4771]: I1011 10:27:52.061149 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} err="failed to get container status \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": rpc error: code = NotFound desc = could not find container \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": container with ID starting with 676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853 not found: ID does not exist" Oct 11 10:27:52.061226 master-1 kubenswrapper[4771]: I1011 10:27:52.061213 4771 scope.go:117] "RemoveContainer" containerID="7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3" Oct 11 10:27:52.061687 master-1 kubenswrapper[4771]: I1011 10:27:52.061621 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} err="failed to get container status \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": rpc error: code = NotFound desc = could not find container \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": container with ID starting with 7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3 not found: ID does not exist" Oct 11 10:27:52.061687 master-1 kubenswrapper[4771]: I1011 10:27:52.061656 4771 scope.go:117] "RemoveContainer" containerID="ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe" Oct 11 10:27:52.065453 master-1 kubenswrapper[4771]: I1011 10:27:52.063181 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} err="failed to get container status \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": rpc error: code = NotFound desc = could not find container \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": container with ID starting with ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe not found: ID does not exist" Oct 11 10:27:52.065453 master-1 kubenswrapper[4771]: I1011 10:27:52.063273 4771 scope.go:117] "RemoveContainer" containerID="cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0" Oct 11 10:27:52.065453 master-1 kubenswrapper[4771]: I1011 10:27:52.063892 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} err="failed to get container status \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": rpc error: code = NotFound desc = could not find container \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": container with ID starting with cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0 not found: ID does not exist" Oct 11 10:27:52.065453 master-1 kubenswrapper[4771]: I1011 10:27:52.063960 4771 scope.go:117] "RemoveContainer" containerID="fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79" Oct 11 10:27:52.065453 master-1 kubenswrapper[4771]: I1011 10:27:52.064498 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79"} err="failed to get container status \"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79\": rpc error: code = NotFound desc = could not find container \"fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79\": container with ID starting with fd77afbe7b6353ee8dcc04d725d33a007170c03c64f8511748463b5783682a79 not found: ID does not exist" Oct 11 10:27:52.065453 master-1 kubenswrapper[4771]: I1011 10:27:52.064561 4771 scope.go:117] "RemoveContainer" containerID="2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88" Oct 11 10:27:52.065453 master-1 kubenswrapper[4771]: I1011 10:27:52.065012 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88"} err="failed to get container status \"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88\": rpc error: code = NotFound desc = could not find container \"2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88\": container with ID starting with 2708e75524558fa1175bcbd5968377f34b9b9be54108180f4fce32d42c460d88 not found: ID does not exist" Oct 11 10:27:52.065453 master-1 kubenswrapper[4771]: I1011 10:27:52.065103 4771 scope.go:117] "RemoveContainer" containerID="92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce" Oct 11 10:27:52.065946 master-1 kubenswrapper[4771]: I1011 10:27:52.065582 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce"} err="failed to get container status \"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce\": rpc error: code = NotFound desc = could not find container \"92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce\": container with ID starting with 92aa93b94b03322d20f3f2cf32f51c2dcd28d35b3f0970cbe4a502bda9e364ce not found: ID does not exist" Oct 11 10:27:52.065946 master-1 kubenswrapper[4771]: I1011 10:27:52.065632 4771 scope.go:117] "RemoveContainer" containerID="2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc" Oct 11 10:27:52.067342 master-1 kubenswrapper[4771]: I1011 10:27:52.066639 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc"} err="failed to get container status \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": rpc error: code = NotFound desc = could not find container \"2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc\": container with ID starting with 2f11f85953c7cb2988ced62308f447053945264a4903a23951393dafad255fbc not found: ID does not exist" Oct 11 10:27:52.067342 master-1 kubenswrapper[4771]: I1011 10:27:52.067335 4771 scope.go:117] "RemoveContainer" containerID="0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1" Oct 11 10:27:52.067859 master-1 kubenswrapper[4771]: I1011 10:27:52.067801 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1"} err="failed to get container status \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": rpc error: code = NotFound desc = could not find container \"0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1\": container with ID starting with 0f3a16a6b5a76cbd7451a015fb2dcaa5057fa6c4c5460d1d4b0c3ee2a73197e1 not found: ID does not exist" Oct 11 10:27:52.067859 master-1 kubenswrapper[4771]: I1011 10:27:52.067845 4771 scope.go:117] "RemoveContainer" containerID="676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853" Oct 11 10:27:52.068253 master-1 kubenswrapper[4771]: I1011 10:27:52.068200 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853"} err="failed to get container status \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": rpc error: code = NotFound desc = could not find container \"676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853\": container with ID starting with 676de21f9311cc96340bf12820a36540fd68a571277c4eee1e15c5020bb27853 not found: ID does not exist" Oct 11 10:27:52.068253 master-1 kubenswrapper[4771]: I1011 10:27:52.068241 4771 scope.go:117] "RemoveContainer" containerID="7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3" Oct 11 10:27:52.068761 master-1 kubenswrapper[4771]: I1011 10:27:52.068697 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3"} err="failed to get container status \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": rpc error: code = NotFound desc = could not find container \"7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3\": container with ID starting with 7c23f33a39621d158f13fbb129750170af4f94ff875db3195fd65a2ea659e5e3 not found: ID does not exist" Oct 11 10:27:52.068761 master-1 kubenswrapper[4771]: I1011 10:27:52.068750 4771 scope.go:117] "RemoveContainer" containerID="ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe" Oct 11 10:27:52.069429 master-1 kubenswrapper[4771]: I1011 10:27:52.069333 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe"} err="failed to get container status \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": rpc error: code = NotFound desc = could not find container \"ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe\": container with ID starting with ab7031670dcd4ac8746cd2646283684fcdf9a2065169a55d1845bc4fd97cabfe not found: ID does not exist" Oct 11 10:27:52.069429 master-1 kubenswrapper[4771]: I1011 10:27:52.069418 4771 scope.go:117] "RemoveContainer" containerID="cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0" Oct 11 10:27:52.069974 master-1 kubenswrapper[4771]: I1011 10:27:52.069918 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0"} err="failed to get container status \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": rpc error: code = NotFound desc = could not find container \"cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0\": container with ID starting with cb657d00f3c40cd2473268577084b894fa35c8640de80e8a24d2cc346b3481f0 not found: ID does not exist" Oct 11 10:27:52.436484 master-1 kubenswrapper[4771]: I1011 10:27:52.436255 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:52.436806 master-1 kubenswrapper[4771]: I1011 10:27:52.436451 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:52.437899 master-1 kubenswrapper[4771]: E1011 10:27:52.437815 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:52.438019 master-1 kubenswrapper[4771]: E1011 10:27:52.437941 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:52.444163 master-1 kubenswrapper[4771]: I1011 10:27:52.444104 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96c2d0f1-e436-480c-9e34-9068178f9df4" path="/var/lib/kubelet/pods/96c2d0f1-e436-480c-9e34-9068178f9df4/volumes" Oct 11 10:27:52.856810 master-1 kubenswrapper[4771]: I1011 10:27:52.856711 4771 generic.go:334] "Generic (PLEG): container finished" podID="a199ebda-03a4-4154-902b-28397e4bc616" containerID="e286ed9631e3bc792d7041c1bf8fc3c79727dde398696236177ad7a5a407c619" exitCode=0 Oct 11 10:27:52.856810 master-1 kubenswrapper[4771]: I1011 10:27:52.856760 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerDied","Data":"e286ed9631e3bc792d7041c1bf8fc3c79727dde398696236177ad7a5a407c619"} Oct 11 10:27:53.868565 master-1 kubenswrapper[4771]: I1011 10:27:53.868479 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerStarted","Data":"811acc170d18d1cc1fb2e96244bd657affd5063d11b11fed0eb46a3a7c5a648f"} Oct 11 10:27:53.868565 master-1 kubenswrapper[4771]: I1011 10:27:53.868567 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerStarted","Data":"3c3b5bb1f5483e2ded64735aa85858cf0858f4f695f8c037443c207c64d0f520"} Oct 11 10:27:53.870335 master-1 kubenswrapper[4771]: I1011 10:27:53.868591 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerStarted","Data":"c1d8c24dcdadcad5e7a13a9718b6daeeaaa8375978b55bcf3c855d56d5b71f16"} Oct 11 10:27:53.870335 master-1 kubenswrapper[4771]: I1011 10:27:53.868610 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerStarted","Data":"8627a1f9bab17ac392bffb2dceb0bec6de4b75d5ef5d603424b688f3ef7d9b5f"} Oct 11 10:27:53.870335 master-1 kubenswrapper[4771]: I1011 10:27:53.868628 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerStarted","Data":"dba69f90e730cf13a8b86f0ec5f8fb7feed6a3bdf7e8bd0002e37b08b0d0cab9"} Oct 11 10:27:53.870335 master-1 kubenswrapper[4771]: I1011 10:27:53.868646 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerStarted","Data":"99315b03b176b9548ff6f442300ed5f7301dbc1d41371db939926c6b88f3bbcb"} Oct 11 10:27:54.436581 master-1 kubenswrapper[4771]: I1011 10:27:54.436518 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:54.436850 master-1 kubenswrapper[4771]: E1011 10:27:54.436673 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:54.436976 master-1 kubenswrapper[4771]: I1011 10:27:54.436534 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:54.437283 master-1 kubenswrapper[4771]: E1011 10:27:54.437246 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:56.436332 master-1 kubenswrapper[4771]: I1011 10:27:56.436165 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:56.437255 master-1 kubenswrapper[4771]: I1011 10:27:56.436335 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:56.437255 master-1 kubenswrapper[4771]: E1011 10:27:56.436471 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:56.437255 master-1 kubenswrapper[4771]: E1011 10:27:56.436591 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:56.883617 master-1 kubenswrapper[4771]: I1011 10:27:56.883545 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerStarted","Data":"a31f2faa9277289c626596c01158d31130d2fd07a0622a3dc29355c1c98bcbc4"} Oct 11 10:27:58.437167 master-1 kubenswrapper[4771]: I1011 10:27:58.436692 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:27:58.437944 master-1 kubenswrapper[4771]: E1011 10:27:58.437169 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:27:58.437944 master-1 kubenswrapper[4771]: I1011 10:27:58.437432 4771 scope.go:117] "RemoveContainer" containerID="d6f0bf05ac57d47238297705efd6175b4b0b48e0ab73a222acdf287379d27829" Oct 11 10:27:58.437944 master-1 kubenswrapper[4771]: I1011 10:27:58.437441 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:27:58.437944 master-1 kubenswrapper[4771]: E1011 10:27:58.437703 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-779749f859-5xxzp_openshift-cloud-controller-manager-operator(e115f8be-9e65-4407-8111-568e5ea8ac1b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" podUID="e115f8be-9e65-4407-8111-568e5ea8ac1b" Oct 11 10:27:58.437944 master-1 kubenswrapper[4771]: E1011 10:27:58.437711 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:27:58.894857 master-1 kubenswrapper[4771]: I1011 10:27:58.894799 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" event={"ID":"a199ebda-03a4-4154-902b-28397e4bc616","Type":"ContainerStarted","Data":"ca302a8439820b05b4af2e1ab36feb2534be24776f9c7a40729fdb60938aaa70"} Oct 11 10:27:58.895286 master-1 kubenswrapper[4771]: I1011 10:27:58.895212 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:58.895406 master-1 kubenswrapper[4771]: I1011 10:27:58.895308 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:27:59.899025 master-1 kubenswrapper[4771]: I1011 10:27:59.898915 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:28:00.436822 master-1 kubenswrapper[4771]: I1011 10:28:00.436753 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:28:00.436822 master-1 kubenswrapper[4771]: I1011 10:28:00.436786 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:28:00.437055 master-1 kubenswrapper[4771]: E1011 10:28:00.436944 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:28:00.437096 master-1 kubenswrapper[4771]: E1011 10:28:00.437041 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:28:00.576586 master-1 kubenswrapper[4771]: I1011 10:28:00.576448 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:28:00.576771 master-1 kubenswrapper[4771]: E1011 10:28:00.576697 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 11 10:28:00.576771 master-1 kubenswrapper[4771]: E1011 10:28:00.576736 4771 projected.go:288] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 11 10:28:00.576771 master-1 kubenswrapper[4771]: E1011 10:28:00.576757 4771 projected.go:194] Error preparing data for projected volume kube-api-access-hktrh for pod openshift-network-diagnostics/network-check-target-4pm7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:28:00.576886 master-1 kubenswrapper[4771]: E1011 10:28:00.576839 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh podName:0bde275d-f0a5-4bea-93f7-edd2077e46b4 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:32.576812511 +0000 UTC m=+144.551038982 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hktrh" (UniqueName: "kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh") pod "network-check-target-4pm7x" (UID: "0bde275d-f0a5-4bea-93f7-edd2077e46b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 11 10:28:02.436901 master-1 kubenswrapper[4771]: I1011 10:28:02.436819 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:28:02.437957 master-1 kubenswrapper[4771]: I1011 10:28:02.436914 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:28:02.437957 master-1 kubenswrapper[4771]: E1011 10:28:02.437920 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:28:02.438502 master-1 kubenswrapper[4771]: E1011 10:28:02.438421 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:28:04.436763 master-1 kubenswrapper[4771]: I1011 10:28:04.436270 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:28:04.437543 master-1 kubenswrapper[4771]: I1011 10:28:04.436532 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:28:04.437543 master-1 kubenswrapper[4771]: E1011 10:28:04.436933 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fgjvw" podUID="2c084572-a5c9-4787-8a14-b7d6b0810a1b" Oct 11 10:28:04.437543 master-1 kubenswrapper[4771]: E1011 10:28:04.437090 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4pm7x" podUID="0bde275d-f0a5-4bea-93f7-edd2077e46b4" Oct 11 10:28:04.773351 master-1 kubenswrapper[4771]: I1011 10:28:04.773234 4771 kubelet_node_status.go:724] "Recording event message for node" node="master-1" event="NodeReady" Oct 11 10:28:04.773683 master-1 kubenswrapper[4771]: I1011 10:28:04.773503 4771 kubelet_node_status.go:538] "Fast updating node status as it just became ready" Oct 11 10:28:04.800842 master-1 kubenswrapper[4771]: I1011 10:28:04.800684 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" podStartSLOduration=13.800654259 podStartE2EDuration="13.800654259s" podCreationTimestamp="2025-10-11 10:27:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:27:58.930704738 +0000 UTC m=+110.904931239" watchObservedRunningTime="2025-10-11 10:28:04.800654259 +0000 UTC m=+116.774880780" Oct 11 10:28:04.801809 master-1 kubenswrapper[4771]: I1011 10:28:04.801760 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-t44c5"] Oct 11 10:28:04.802470 master-1 kubenswrapper[4771]: I1011 10:28:04.802398 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:04.806032 master-1 kubenswrapper[4771]: I1011 10:28:04.805952 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-operator"/"iptables-alerter-script" Oct 11 10:28:04.812616 master-1 kubenswrapper[4771]: I1011 10:28:04.812553 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5bt\" (UniqueName: \"kubernetes.io/projected/3346c1b6-593b-4224-802c-25e99e9893a8-kube-api-access-2w5bt\") pod \"iptables-alerter-t44c5\" (UID: \"3346c1b6-593b-4224-802c-25e99e9893a8\") " pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:04.812783 master-1 kubenswrapper[4771]: I1011 10:28:04.812647 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3346c1b6-593b-4224-802c-25e99e9893a8-iptables-alerter-script\") pod \"iptables-alerter-t44c5\" (UID: \"3346c1b6-593b-4224-802c-25e99e9893a8\") " pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:04.812783 master-1 kubenswrapper[4771]: I1011 10:28:04.812689 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3346c1b6-593b-4224-802c-25e99e9893a8-host-slash\") pod \"iptables-alerter-t44c5\" (UID: \"3346c1b6-593b-4224-802c-25e99e9893a8\") " pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:04.913058 master-1 kubenswrapper[4771]: I1011 10:28:04.912949 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5bt\" (UniqueName: \"kubernetes.io/projected/3346c1b6-593b-4224-802c-25e99e9893a8-kube-api-access-2w5bt\") pod \"iptables-alerter-t44c5\" (UID: \"3346c1b6-593b-4224-802c-25e99e9893a8\") " pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:04.913205 master-1 kubenswrapper[4771]: I1011 10:28:04.913073 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3346c1b6-593b-4224-802c-25e99e9893a8-iptables-alerter-script\") pod \"iptables-alerter-t44c5\" (UID: \"3346c1b6-593b-4224-802c-25e99e9893a8\") " pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:04.913205 master-1 kubenswrapper[4771]: I1011 10:28:04.913114 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3346c1b6-593b-4224-802c-25e99e9893a8-host-slash\") pod \"iptables-alerter-t44c5\" (UID: \"3346c1b6-593b-4224-802c-25e99e9893a8\") " pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:04.913205 master-1 kubenswrapper[4771]: I1011 10:28:04.913193 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3346c1b6-593b-4224-802c-25e99e9893a8-host-slash\") pod \"iptables-alerter-t44c5\" (UID: \"3346c1b6-593b-4224-802c-25e99e9893a8\") " pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:04.913929 master-1 kubenswrapper[4771]: I1011 10:28:04.913869 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3346c1b6-593b-4224-802c-25e99e9893a8-iptables-alerter-script\") pod \"iptables-alerter-t44c5\" (UID: \"3346c1b6-593b-4224-802c-25e99e9893a8\") " pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:04.935385 master-1 kubenswrapper[4771]: I1011 10:28:04.935284 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5bt\" (UniqueName: \"kubernetes.io/projected/3346c1b6-593b-4224-802c-25e99e9893a8-kube-api-access-2w5bt\") pod \"iptables-alerter-t44c5\" (UID: \"3346c1b6-593b-4224-802c-25e99e9893a8\") " pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:05.126933 master-1 kubenswrapper[4771]: I1011 10:28:05.126790 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t44c5" Oct 11 10:28:05.150658 master-1 kubenswrapper[4771]: W1011 10:28:05.150586 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3346c1b6_593b_4224_802c_25e99e9893a8.slice/crio-47eb04e21cc8cca54dd2af469c2901eb6aaa71fb7ab0ddf3d031a884da55e0f6 WatchSource:0}: Error finding container 47eb04e21cc8cca54dd2af469c2901eb6aaa71fb7ab0ddf3d031a884da55e0f6: Status 404 returned error can't find the container with id 47eb04e21cc8cca54dd2af469c2901eb6aaa71fb7ab0ddf3d031a884da55e0f6 Oct 11 10:28:05.921378 master-1 kubenswrapper[4771]: I1011 10:28:05.921274 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t44c5" event={"ID":"3346c1b6-593b-4224-802c-25e99e9893a8","Type":"ContainerStarted","Data":"47eb04e21cc8cca54dd2af469c2901eb6aaa71fb7ab0ddf3d031a884da55e0f6"} Oct 11 10:28:06.436619 master-1 kubenswrapper[4771]: I1011 10:28:06.436479 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:28:06.436921 master-1 kubenswrapper[4771]: I1011 10:28:06.436775 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:28:06.439236 master-1 kubenswrapper[4771]: I1011 10:28:06.439172 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-diagnostics"/"kube-root-ca.crt" Oct 11 10:28:06.440006 master-1 kubenswrapper[4771]: I1011 10:28:06.439929 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-diagnostics"/"openshift-service-ca.crt" Oct 11 10:28:06.440268 master-1 kubenswrapper[4771]: I1011 10:28:06.440197 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-multus"/"metrics-daemon-secret" Oct 11 10:28:10.437425 master-1 kubenswrapper[4771]: I1011 10:28:10.437296 4771 scope.go:117] "RemoveContainer" containerID="d6f0bf05ac57d47238297705efd6175b4b0b48e0ab73a222acdf287379d27829" Oct 11 10:28:10.940980 master-1 kubenswrapper[4771]: I1011 10:28:10.940768 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/3.log" Oct 11 10:28:10.942080 master-1 kubenswrapper[4771]: I1011 10:28:10.942019 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/2.log" Oct 11 10:28:10.943104 master-1 kubenswrapper[4771]: I1011 10:28:10.943050 4771 generic.go:334] "Generic (PLEG): container finished" podID="e115f8be-9e65-4407-8111-568e5ea8ac1b" containerID="793c72629ffb5d64763cce906980f11774530f02d707e0389b69155b33560c5d" exitCode=1 Oct 11 10:28:10.943164 master-1 kubenswrapper[4771]: I1011 10:28:10.943116 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" event={"ID":"e115f8be-9e65-4407-8111-568e5ea8ac1b","Type":"ContainerDied","Data":"793c72629ffb5d64763cce906980f11774530f02d707e0389b69155b33560c5d"} Oct 11 10:28:10.943209 master-1 kubenswrapper[4771]: I1011 10:28:10.943189 4771 scope.go:117] "RemoveContainer" containerID="d6f0bf05ac57d47238297705efd6175b4b0b48e0ab73a222acdf287379d27829" Oct 11 10:28:10.944000 master-1 kubenswrapper[4771]: I1011 10:28:10.943938 4771 scope.go:117] "RemoveContainer" containerID="793c72629ffb5d64763cce906980f11774530f02d707e0389b69155b33560c5d" Oct 11 10:28:10.944234 master-1 kubenswrapper[4771]: E1011 10:28:10.944184 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-779749f859-5xxzp_openshift-cloud-controller-manager-operator(e115f8be-9e65-4407-8111-568e5ea8ac1b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" podUID="e115f8be-9e65-4407-8111-568e5ea8ac1b" Oct 11 10:28:11.791780 master-1 kubenswrapper[4771]: I1011 10:28:11.791696 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:28:11.793653 master-1 kubenswrapper[4771]: I1011 10:28:11.793596 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:28:11.811569 master-1 kubenswrapper[4771]: I1011 10:28:11.811487 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p9l4v" Oct 11 10:28:11.949312 master-1 kubenswrapper[4771]: I1011 10:28:11.949206 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/3.log" Oct 11 10:28:13.959198 master-1 kubenswrapper[4771]: I1011 10:28:13.959070 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t44c5" event={"ID":"3346c1b6-593b-4224-802c-25e99e9893a8","Type":"ContainerStarted","Data":"80ee69bb8bb9ee41ed409fb0049d311eb9b31c6f3c980d9975c6a0d160195a6d"} Oct 11 10:28:13.977490 master-1 kubenswrapper[4771]: I1011 10:28:13.977331 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t44c5" podStartSLOduration=5.91420538 podStartE2EDuration="9.977306523s" podCreationTimestamp="2025-10-11 10:28:04 +0000 UTC" firstStartedPulling="2025-10-11 10:28:05.15416437 +0000 UTC m=+117.128390851" lastFinishedPulling="2025-10-11 10:28:09.217265513 +0000 UTC m=+121.191491994" observedRunningTime="2025-10-11 10:28:13.975242798 +0000 UTC m=+125.949469279" watchObservedRunningTime="2025-10-11 10:28:13.977306523 +0000 UTC m=+125.951533004" Oct 11 10:28:17.927932 master-1 kubenswrapper[4771]: I1011 10:28:17.927814 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:28:17.931209 master-1 kubenswrapper[4771]: I1011 10:28:17.931150 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-multus"/"metrics-daemon-secret" Oct 11 10:28:17.938899 master-1 kubenswrapper[4771]: E1011 10:28:17.938832 4771 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 11 10:28:17.938899 master-1 kubenswrapper[4771]: E1011 10:28:17.938895 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs podName:2c084572-a5c9-4787-8a14-b7d6b0810a1b nodeName:}" failed. No retries permitted until 2025-10-11 10:29:21.938877361 +0000 UTC m=+193.913103812 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs") pod "network-metrics-daemon-fgjvw" (UID: "2c084572-a5c9-4787-8a14-b7d6b0810a1b") : secret "metrics-daemon-secret" not found Oct 11 10:28:18.041803 master-1 kubenswrapper[4771]: I1011 10:28:18.041680 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["assisted-installer/master-1-debug-vwkqm"] Oct 11 10:28:18.042110 master-1 kubenswrapper[4771]: I1011 10:28:18.042042 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:18.045041 master-1 kubenswrapper[4771]: I1011 10:28:18.044986 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"assisted-installer"/"openshift-service-ca.crt" Oct 11 10:28:18.045897 master-1 kubenswrapper[4771]: I1011 10:28:18.045861 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"assisted-installer"/"kube-root-ca.crt" Oct 11 10:28:18.129637 master-1 kubenswrapper[4771]: I1011 10:28:18.129562 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254q5\" (UniqueName: \"kubernetes.io/projected/e5877eb5-69db-40eb-af0c-096d52c3fc4d-kube-api-access-254q5\") pod \"master-1-debug-vwkqm\" (UID: \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\") " pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:18.129909 master-1 kubenswrapper[4771]: I1011 10:28:18.129633 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5877eb5-69db-40eb-af0c-096d52c3fc4d-host\") pod \"master-1-debug-vwkqm\" (UID: \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\") " pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:18.231112 master-1 kubenswrapper[4771]: I1011 10:28:18.230935 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-254q5\" (UniqueName: \"kubernetes.io/projected/e5877eb5-69db-40eb-af0c-096d52c3fc4d-kube-api-access-254q5\") pod \"master-1-debug-vwkqm\" (UID: \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\") " pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:18.231112 master-1 kubenswrapper[4771]: I1011 10:28:18.230992 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5877eb5-69db-40eb-af0c-096d52c3fc4d-host\") pod \"master-1-debug-vwkqm\" (UID: \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\") " pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:18.231112 master-1 kubenswrapper[4771]: I1011 10:28:18.231074 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5877eb5-69db-40eb-af0c-096d52c3fc4d-host\") pod \"master-1-debug-vwkqm\" (UID: \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\") " pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:18.253972 master-1 kubenswrapper[4771]: I1011 10:28:18.253817 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-254q5\" (UniqueName: \"kubernetes.io/projected/e5877eb5-69db-40eb-af0c-096d52c3fc4d-kube-api-access-254q5\") pod \"master-1-debug-vwkqm\" (UID: \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\") " pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:18.281740 master-1 kubenswrapper[4771]: I1011 10:28:18.281654 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92"] Oct 11 10:28:18.282240 master-1 kubenswrapper[4771]: I1011 10:28:18.282199 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92" Oct 11 10:28:18.284674 master-1 kubenswrapper[4771]: I1011 10:28:18.284640 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-storage-version-migrator"/"openshift-service-ca.crt" Oct 11 10:28:18.284942 master-1 kubenswrapper[4771]: I1011 10:28:18.284903 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-storage-version-migrator"/"kube-root-ca.crt" Oct 11 10:28:18.291339 master-1 kubenswrapper[4771]: I1011 10:28:18.291285 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92"] Oct 11 10:28:18.331769 master-1 kubenswrapper[4771]: I1011 10:28:18.331714 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrs8\" (UniqueName: \"kubernetes.io/projected/b62583d2-1a1e-44fd-871e-1c48e3cb1732-kube-api-access-lvrs8\") pod \"migrator-d8c4d9469-bxq92\" (UID: \"b62583d2-1a1e-44fd-871e-1c48e3cb1732\") " pod="openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92" Oct 11 10:28:18.375139 master-1 kubenswrapper[4771]: I1011 10:28:18.375053 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:18.394024 master-1 kubenswrapper[4771]: W1011 10:28:18.393955 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5877eb5_69db_40eb_af0c_096d52c3fc4d.slice/crio-78e52134fc9902f7609237c0f679b9c5002f127b8ab80d9c3907b1e5458d8504 WatchSource:0}: Error finding container 78e52134fc9902f7609237c0f679b9c5002f127b8ab80d9c3907b1e5458d8504: Status 404 returned error can't find the container with id 78e52134fc9902f7609237c0f679b9c5002f127b8ab80d9c3907b1e5458d8504 Oct 11 10:28:18.432316 master-1 kubenswrapper[4771]: I1011 10:28:18.432250 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrs8\" (UniqueName: \"kubernetes.io/projected/b62583d2-1a1e-44fd-871e-1c48e3cb1732-kube-api-access-lvrs8\") pod \"migrator-d8c4d9469-bxq92\" (UID: \"b62583d2-1a1e-44fd-871e-1c48e3cb1732\") " pod="openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92" Oct 11 10:28:18.455760 master-1 kubenswrapper[4771]: I1011 10:28:18.455668 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrs8\" (UniqueName: \"kubernetes.io/projected/b62583d2-1a1e-44fd-871e-1c48e3cb1732-kube-api-access-lvrs8\") pod \"migrator-d8c4d9469-bxq92\" (UID: \"b62583d2-1a1e-44fd-871e-1c48e3cb1732\") " pod="openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92" Oct 11 10:28:18.598674 master-1 kubenswrapper[4771]: I1011 10:28:18.598594 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92" Oct 11 10:28:18.831833 master-1 kubenswrapper[4771]: I1011 10:28:18.831418 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m"] Oct 11 10:28:18.832222 master-1 kubenswrapper[4771]: I1011 10:28:18.832183 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m" Oct 11 10:28:18.835376 master-1 kubenswrapper[4771]: I1011 10:28:18.835325 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cluster-storage-operator"/"openshift-service-ca.crt" Oct 11 10:28:18.835956 master-1 kubenswrapper[4771]: I1011 10:28:18.835921 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cluster-storage-operator"/"kube-root-ca.crt" Oct 11 10:28:18.838331 master-1 kubenswrapper[4771]: I1011 10:28:18.838297 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m"] Oct 11 10:28:18.867055 master-1 kubenswrapper[4771]: I1011 10:28:18.866799 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92"] Oct 11 10:28:18.936542 master-1 kubenswrapper[4771]: I1011 10:28:18.936479 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7plbp\" (UniqueName: \"kubernetes.io/projected/09cf0cd5-a6f6-4b35-88cf-ca6ca4402656-kube-api-access-7plbp\") pod \"csi-snapshot-controller-ddd7d64cd-c2t4m\" (UID: \"09cf0cd5-a6f6-4b35-88cf-ca6ca4402656\") " pod="openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m" Oct 11 10:28:18.976290 master-1 kubenswrapper[4771]: I1011 10:28:18.976209 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92" event={"ID":"b62583d2-1a1e-44fd-871e-1c48e3cb1732","Type":"ContainerStarted","Data":"d98dba785d03bcc465a43f2068692663b1667d84f0433b1551e2ef33f927aca9"} Oct 11 10:28:18.977627 master-1 kubenswrapper[4771]: I1011 10:28:18.977571 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="assisted-installer/master-1-debug-vwkqm" event={"ID":"e5877eb5-69db-40eb-af0c-096d52c3fc4d","Type":"ContainerStarted","Data":"78e52134fc9902f7609237c0f679b9c5002f127b8ab80d9c3907b1e5458d8504"} Oct 11 10:28:19.037481 master-1 kubenswrapper[4771]: I1011 10:28:19.037349 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-7plbp\" (UniqueName: \"kubernetes.io/projected/09cf0cd5-a6f6-4b35-88cf-ca6ca4402656-kube-api-access-7plbp\") pod \"csi-snapshot-controller-ddd7d64cd-c2t4m\" (UID: \"09cf0cd5-a6f6-4b35-88cf-ca6ca4402656\") " pod="openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m" Oct 11 10:28:19.073300 master-1 kubenswrapper[4771]: I1011 10:28:19.073150 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-7plbp\" (UniqueName: \"kubernetes.io/projected/09cf0cd5-a6f6-4b35-88cf-ca6ca4402656-kube-api-access-7plbp\") pod \"csi-snapshot-controller-ddd7d64cd-c2t4m\" (UID: \"09cf0cd5-a6f6-4b35-88cf-ca6ca4402656\") " pod="openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m" Oct 11 10:28:19.146038 master-1 kubenswrapper[4771]: I1011 10:28:19.145875 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m" Oct 11 10:28:19.421498 master-1 kubenswrapper[4771]: I1011 10:28:19.421328 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m"] Oct 11 10:28:19.431382 master-1 kubenswrapper[4771]: W1011 10:28:19.431283 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09cf0cd5_a6f6_4b35_88cf_ca6ca4402656.slice/crio-8c76b004f57f73a0758ab97b4b322aadaddacefb450c5c506970c872696e5d88 WatchSource:0}: Error finding container 8c76b004f57f73a0758ab97b4b322aadaddacefb450c5c506970c872696e5d88: Status 404 returned error can't find the container with id 8c76b004f57f73a0758ab97b4b322aadaddacefb450c5c506970c872696e5d88 Oct 11 10:28:19.982100 master-1 kubenswrapper[4771]: I1011 10:28:19.982048 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m" event={"ID":"09cf0cd5-a6f6-4b35-88cf-ca6ca4402656","Type":"ContainerStarted","Data":"8c76b004f57f73a0758ab97b4b322aadaddacefb450c5c506970c872696e5d88"} Oct 11 10:28:20.708535 master-1 kubenswrapper[4771]: I1011 10:28:20.708335 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-64446499c7-sb6sm"] Oct 11 10:28:20.708747 master-1 kubenswrapper[4771]: I1011 10:28:20.708728 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.712077 master-1 kubenswrapper[4771]: I1011 10:28:20.711627 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-service-ca"/"signing-key" Oct 11 10:28:20.712077 master-1 kubenswrapper[4771]: I1011 10:28:20.711753 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-service-ca"/"signing-cabundle" Oct 11 10:28:20.712077 master-1 kubenswrapper[4771]: I1011 10:28:20.711771 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-service-ca"/"openshift-service-ca.crt" Oct 11 10:28:20.712077 master-1 kubenswrapper[4771]: I1011 10:28:20.711941 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-service-ca"/"kube-root-ca.crt" Oct 11 10:28:20.720037 master-1 kubenswrapper[4771]: I1011 10:28:20.719974 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-64446499c7-sb6sm"] Oct 11 10:28:20.753017 master-1 kubenswrapper[4771]: I1011 10:28:20.752813 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/27c04f6d-d04c-41b4-bcaf-19edb41f6604-signing-cabundle\") pod \"service-ca-64446499c7-sb6sm\" (UID: \"27c04f6d-d04c-41b4-bcaf-19edb41f6604\") " pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.753017 master-1 kubenswrapper[4771]: I1011 10:28:20.752893 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/27c04f6d-d04c-41b4-bcaf-19edb41f6604-signing-key\") pod \"service-ca-64446499c7-sb6sm\" (UID: \"27c04f6d-d04c-41b4-bcaf-19edb41f6604\") " pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.753017 master-1 kubenswrapper[4771]: I1011 10:28:20.752960 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6pz9\" (UniqueName: \"kubernetes.io/projected/27c04f6d-d04c-41b4-bcaf-19edb41f6604-kube-api-access-t6pz9\") pod \"service-ca-64446499c7-sb6sm\" (UID: \"27c04f6d-d04c-41b4-bcaf-19edb41f6604\") " pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.854659 master-1 kubenswrapper[4771]: I1011 10:28:20.853877 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-t6pz9\" (UniqueName: \"kubernetes.io/projected/27c04f6d-d04c-41b4-bcaf-19edb41f6604-kube-api-access-t6pz9\") pod \"service-ca-64446499c7-sb6sm\" (UID: \"27c04f6d-d04c-41b4-bcaf-19edb41f6604\") " pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.854659 master-1 kubenswrapper[4771]: I1011 10:28:20.853998 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/27c04f6d-d04c-41b4-bcaf-19edb41f6604-signing-cabundle\") pod \"service-ca-64446499c7-sb6sm\" (UID: \"27c04f6d-d04c-41b4-bcaf-19edb41f6604\") " pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.854659 master-1 kubenswrapper[4771]: I1011 10:28:20.854028 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/27c04f6d-d04c-41b4-bcaf-19edb41f6604-signing-key\") pod \"service-ca-64446499c7-sb6sm\" (UID: \"27c04f6d-d04c-41b4-bcaf-19edb41f6604\") " pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.855659 master-1 kubenswrapper[4771]: I1011 10:28:20.855597 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/27c04f6d-d04c-41b4-bcaf-19edb41f6604-signing-cabundle\") pod \"service-ca-64446499c7-sb6sm\" (UID: \"27c04f6d-d04c-41b4-bcaf-19edb41f6604\") " pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.862318 master-1 kubenswrapper[4771]: I1011 10:28:20.862258 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/27c04f6d-d04c-41b4-bcaf-19edb41f6604-signing-key\") pod \"service-ca-64446499c7-sb6sm\" (UID: \"27c04f6d-d04c-41b4-bcaf-19edb41f6604\") " pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.872611 master-1 kubenswrapper[4771]: I1011 10:28:20.872549 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6pz9\" (UniqueName: \"kubernetes.io/projected/27c04f6d-d04c-41b4-bcaf-19edb41f6604-kube-api-access-t6pz9\") pod \"service-ca-64446499c7-sb6sm\" (UID: \"27c04f6d-d04c-41b4-bcaf-19edb41f6604\") " pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:20.985803 master-1 kubenswrapper[4771]: I1011 10:28:20.985675 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92" event={"ID":"b62583d2-1a1e-44fd-871e-1c48e3cb1732","Type":"ContainerStarted","Data":"9dd1ab667102e62c18c0e52de9fa777b29d04f09335b1e34df94c64f0b365c01"} Oct 11 10:28:20.985803 master-1 kubenswrapper[4771]: I1011 10:28:20.985736 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92" event={"ID":"b62583d2-1a1e-44fd-871e-1c48e3cb1732","Type":"ContainerStarted","Data":"6c4a2bd203d96b682cdd610dfa28d37a6f351ee6104d4c8e1fb079f0631ff99e"} Oct 11 10:28:20.999807 master-1 kubenswrapper[4771]: I1011 10:28:20.999520 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d8c4d9469-bxq92" podStartSLOduration=1.769219514 podStartE2EDuration="2.999503822s" podCreationTimestamp="2025-10-11 10:28:18 +0000 UTC" firstStartedPulling="2025-10-11 10:28:18.882593455 +0000 UTC m=+130.856819896" lastFinishedPulling="2025-10-11 10:28:20.112877763 +0000 UTC m=+132.087104204" observedRunningTime="2025-10-11 10:28:20.999158543 +0000 UTC m=+132.973385044" watchObservedRunningTime="2025-10-11 10:28:20.999503822 +0000 UTC m=+132.973730273" Oct 11 10:28:21.023236 master-1 kubenswrapper[4771]: I1011 10:28:21.023192 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-64446499c7-sb6sm" Oct 11 10:28:23.437681 master-1 kubenswrapper[4771]: I1011 10:28:23.437246 4771 scope.go:117] "RemoveContainer" containerID="793c72629ffb5d64763cce906980f11774530f02d707e0389b69155b33560c5d" Oct 11 10:28:23.438759 master-1 kubenswrapper[4771]: E1011 10:28:23.437843 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-779749f859-5xxzp_openshift-cloud-controller-manager-operator(e115f8be-9e65-4407-8111-568e5ea8ac1b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" podUID="e115f8be-9e65-4407-8111-568e5ea8ac1b" Oct 11 10:28:24.937682 master-1 kubenswrapper[4771]: I1011 10:28:24.937344 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-64446499c7-sb6sm"] Oct 11 10:28:24.948924 master-1 kubenswrapper[4771]: W1011 10:28:24.948894 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c04f6d_d04c_41b4_bcaf_19edb41f6604.slice/crio-6ae068076842dd077497e3ab2fe775738f4feeaf33e9d0270c74c49a5c55b246 WatchSource:0}: Error finding container 6ae068076842dd077497e3ab2fe775738f4feeaf33e9d0270c74c49a5c55b246: Status 404 returned error can't find the container with id 6ae068076842dd077497e3ab2fe775738f4feeaf33e9d0270c74c49a5c55b246 Oct 11 10:28:24.998588 master-1 kubenswrapper[4771]: I1011 10:28:24.998439 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m" event={"ID":"09cf0cd5-a6f6-4b35-88cf-ca6ca4402656","Type":"ContainerStarted","Data":"93e6fb561deb65d2eaccfde1779f8f7f72d30de6e99e33cb683f39fa4720cc36"} Oct 11 10:28:24.999815 master-1 kubenswrapper[4771]: I1011 10:28:24.999745 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-64446499c7-sb6sm" event={"ID":"27c04f6d-d04c-41b4-bcaf-19edb41f6604","Type":"ContainerStarted","Data":"6ae068076842dd077497e3ab2fe775738f4feeaf33e9d0270c74c49a5c55b246"} Oct 11 10:28:25.001075 master-1 kubenswrapper[4771]: I1011 10:28:25.000992 4771 generic.go:334] "Generic (PLEG): container finished" podID="e5877eb5-69db-40eb-af0c-096d52c3fc4d" containerID="723577cf73aef3a8a2e4442f397b70704c28142936c48024dbef159c3112ef95" exitCode=0 Oct 11 10:28:25.001075 master-1 kubenswrapper[4771]: I1011 10:28:25.001037 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="assisted-installer/master-1-debug-vwkqm" event={"ID":"e5877eb5-69db-40eb-af0c-096d52c3fc4d","Type":"ContainerDied","Data":"723577cf73aef3a8a2e4442f397b70704c28142936c48024dbef159c3112ef95"} Oct 11 10:28:25.016470 master-1 kubenswrapper[4771]: I1011 10:28:25.016334 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/csi-snapshot-controller-ddd7d64cd-c2t4m" podStartSLOduration=1.698926228 podStartE2EDuration="7.016311315s" podCreationTimestamp="2025-10-11 10:28:18 +0000 UTC" firstStartedPulling="2025-10-11 10:28:19.43482432 +0000 UTC m=+131.409050761" lastFinishedPulling="2025-10-11 10:28:24.752209377 +0000 UTC m=+136.726435848" observedRunningTime="2025-10-11 10:28:25.01384317 +0000 UTC m=+136.988069611" watchObservedRunningTime="2025-10-11 10:28:25.016311315 +0000 UTC m=+136.990537796" Oct 11 10:28:25.056789 master-1 kubenswrapper[4771]: I1011 10:28:25.056698 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["assisted-installer/master-1-debug-vwkqm"] Oct 11 10:28:25.058142 master-1 kubenswrapper[4771]: I1011 10:28:25.058077 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["assisted-installer/master-1-debug-vwkqm"] Oct 11 10:28:26.040708 master-1 kubenswrapper[4771]: I1011 10:28:26.040635 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:26.100405 master-1 kubenswrapper[4771]: I1011 10:28:26.100300 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5877eb5-69db-40eb-af0c-096d52c3fc4d-host\") pod \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\" (UID: \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\") " Oct 11 10:28:26.100693 master-1 kubenswrapper[4771]: I1011 10:28:26.100459 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-254q5\" (UniqueName: \"kubernetes.io/projected/e5877eb5-69db-40eb-af0c-096d52c3fc4d-kube-api-access-254q5\") pod \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\" (UID: \"e5877eb5-69db-40eb-af0c-096d52c3fc4d\") " Oct 11 10:28:26.100693 master-1 kubenswrapper[4771]: I1011 10:28:26.100491 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e5877eb5-69db-40eb-af0c-096d52c3fc4d-host" (OuterVolumeSpecName: "host") pod "e5877eb5-69db-40eb-af0c-096d52c3fc4d" (UID: "e5877eb5-69db-40eb-af0c-096d52c3fc4d"). InnerVolumeSpecName "host". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:28:26.100693 master-1 kubenswrapper[4771]: I1011 10:28:26.100680 4771 reconciler_common.go:293] "Volume detached for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5877eb5-69db-40eb-af0c-096d52c3fc4d-host\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:26.107284 master-1 kubenswrapper[4771]: I1011 10:28:26.107186 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5877eb5-69db-40eb-af0c-096d52c3fc4d-kube-api-access-254q5" (OuterVolumeSpecName: "kube-api-access-254q5") pod "e5877eb5-69db-40eb-af0c-096d52c3fc4d" (UID: "e5877eb5-69db-40eb-af0c-096d52c3fc4d"). InnerVolumeSpecName "kube-api-access-254q5". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:28:26.201640 master-1 kubenswrapper[4771]: I1011 10:28:26.201554 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-254q5\" (UniqueName: \"kubernetes.io/projected/e5877eb5-69db-40eb-af0c-096d52c3fc4d-kube-api-access-254q5\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:26.442330 master-1 kubenswrapper[4771]: I1011 10:28:26.442255 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5877eb5-69db-40eb-af0c-096d52c3fc4d" path="/var/lib/kubelet/pods/e5877eb5-69db-40eb-af0c-096d52c3fc4d/volumes" Oct 11 10:28:27.009267 master-1 kubenswrapper[4771]: I1011 10:28:27.008670 4771 scope.go:117] "RemoveContainer" containerID="723577cf73aef3a8a2e4442f397b70704c28142936c48024dbef159c3112ef95" Oct 11 10:28:27.009267 master-1 kubenswrapper[4771]: I1011 10:28:27.008746 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="assisted-installer/master-1-debug-vwkqm" Oct 11 10:28:27.011488 master-1 kubenswrapper[4771]: I1011 10:28:27.010857 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-64446499c7-sb6sm" event={"ID":"27c04f6d-d04c-41b4-bcaf-19edb41f6604","Type":"ContainerStarted","Data":"2ce7b303d28a1305e8fa0a23a119002db3a3528b142e62f30f20198e8cfc40c9"} Oct 11 10:28:27.025317 master-1 kubenswrapper[4771]: I1011 10:28:27.024980 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-64446499c7-sb6sm" podStartSLOduration=5.116819782 podStartE2EDuration="7.024956518s" podCreationTimestamp="2025-10-11 10:28:20 +0000 UTC" firstStartedPulling="2025-10-11 10:28:24.951011322 +0000 UTC m=+136.925237763" lastFinishedPulling="2025-10-11 10:28:26.859148038 +0000 UTC m=+138.833374499" observedRunningTime="2025-10-11 10:28:27.023019147 +0000 UTC m=+138.997245598" watchObservedRunningTime="2025-10-11 10:28:27.024956518 +0000 UTC m=+138.999182999" Oct 11 10:28:30.053712 master-1 kubenswrapper[4771]: I1011 10:28:30.053621 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-apiserver/apiserver-796c687c6d-9b677"] Oct 11 10:28:30.054753 master-1 kubenswrapper[4771]: E1011 10:28:30.054726 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e5877eb5-69db-40eb-af0c-096d52c3fc4d" containerName="container-00" Oct 11 10:28:30.054906 master-1 kubenswrapper[4771]: I1011 10:28:30.054889 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5877eb5-69db-40eb-af0c-096d52c3fc4d" containerName="container-00" Oct 11 10:28:30.055049 master-1 kubenswrapper[4771]: I1011 10:28:30.055033 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e5877eb5-69db-40eb-af0c-096d52c3fc4d" containerName="container-00" Oct 11 10:28:30.055679 master-1 kubenswrapper[4771]: I1011 10:28:30.055650 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.059831 master-1 kubenswrapper[4771]: I1011 10:28:30.059441 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"etcd-client" Oct 11 10:28:30.059831 master-1 kubenswrapper[4771]: I1011 10:28:30.059628 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"serving-cert" Oct 11 10:28:30.059831 master-1 kubenswrapper[4771]: I1011 10:28:30.059690 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"etcd-serving-ca" Oct 11 10:28:30.059831 master-1 kubenswrapper[4771]: I1011 10:28:30.059633 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"kube-root-ca.crt" Oct 11 10:28:30.060136 master-1 kubenswrapper[4771]: I1011 10:28:30.059914 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"image-import-ca" Oct 11 10:28:30.064302 master-1 kubenswrapper[4771]: I1011 10:28:30.061300 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"audit-0" Oct 11 10:28:30.066084 master-1 kubenswrapper[4771]: I1011 10:28:30.065110 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"encryption-config-0" Oct 11 10:28:30.066499 master-1 kubenswrapper[4771]: I1011 10:28:30.066166 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"config" Oct 11 10:28:30.067068 master-1 kubenswrapper[4771]: I1011 10:28:30.066564 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"openshift-service-ca.crt" Oct 11 10:28:30.078570 master-1 kubenswrapper[4771]: I1011 10:28:30.078505 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-apiserver/apiserver-796c687c6d-9b677"] Oct 11 10:28:30.096426 master-1 kubenswrapper[4771]: I1011 10:28:30.095965 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"trusted-ca-bundle" Oct 11 10:28:30.138226 master-1 kubenswrapper[4771]: I1011 10:28:30.137943 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-client\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138226 master-1 kubenswrapper[4771]: I1011 10:28:30.138020 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138226 master-1 kubenswrapper[4771]: I1011 10:28:30.138060 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-config\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138226 master-1 kubenswrapper[4771]: I1011 10:28:30.138099 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-encryption-config\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138226 master-1 kubenswrapper[4771]: I1011 10:28:30.138174 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit-dir\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138226 master-1 kubenswrapper[4771]: I1011 10:28:30.138237 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-image-import-ca\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138901 master-1 kubenswrapper[4771]: I1011 10:28:30.138308 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-node-pullsecrets\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138901 master-1 kubenswrapper[4771]: I1011 10:28:30.138349 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-trusted-ca-bundle\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138901 master-1 kubenswrapper[4771]: I1011 10:28:30.138421 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-serving-cert\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138901 master-1 kubenswrapper[4771]: I1011 10:28:30.138480 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-serving-ca\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.138901 master-1 kubenswrapper[4771]: I1011 10:28:30.138516 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkqb\" (UniqueName: \"kubernetes.io/projected/94d811f4-4ac9-46b0-b937-d3370b1b4305-kube-api-access-dbkqb\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239558 master-1 kubenswrapper[4771]: I1011 10:28:30.239427 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-config\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239558 master-1 kubenswrapper[4771]: I1011 10:28:30.239541 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-encryption-config\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239993 master-1 kubenswrapper[4771]: I1011 10:28:30.239604 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit-dir\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239993 master-1 kubenswrapper[4771]: I1011 10:28:30.239644 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-image-import-ca\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239993 master-1 kubenswrapper[4771]: I1011 10:28:30.239708 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-node-pullsecrets\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239993 master-1 kubenswrapper[4771]: I1011 10:28:30.239745 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-serving-cert\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239993 master-1 kubenswrapper[4771]: I1011 10:28:30.239775 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-trusted-ca-bundle\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239993 master-1 kubenswrapper[4771]: I1011 10:28:30.239833 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-serving-ca\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239993 master-1 kubenswrapper[4771]: I1011 10:28:30.239835 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit-dir\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239993 master-1 kubenswrapper[4771]: I1011 10:28:30.239866 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkqb\" (UniqueName: \"kubernetes.io/projected/94d811f4-4ac9-46b0-b937-d3370b1b4305-kube-api-access-dbkqb\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.239993 master-1 kubenswrapper[4771]: I1011 10:28:30.239952 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-node-pullsecrets\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.240776 master-1 kubenswrapper[4771]: I1011 10:28:30.240008 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-client\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.240776 master-1 kubenswrapper[4771]: I1011 10:28:30.240097 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.240776 master-1 kubenswrapper[4771]: E1011 10:28:30.240235 4771 configmap.go:193] Couldn't get configMap openshift-apiserver/audit-0: configmap "audit-0" not found Oct 11 10:28:30.240776 master-1 kubenswrapper[4771]: E1011 10:28:30.240345 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit podName:94d811f4-4ac9-46b0-b937-d3370b1b4305 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:30.740306693 +0000 UTC m=+142.714533164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "audit" (UniqueName: "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit") pod "apiserver-796c687c6d-9b677" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305") : configmap "audit-0" not found Oct 11 10:28:30.241347 master-1 kubenswrapper[4771]: I1011 10:28:30.241257 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-config\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.241883 master-1 kubenswrapper[4771]: I1011 10:28:30.241814 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-image-import-ca\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.242230 master-1 kubenswrapper[4771]: I1011 10:28:30.242112 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-serving-ca\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.242499 master-1 kubenswrapper[4771]: I1011 10:28:30.242451 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-trusted-ca-bundle\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.247163 master-1 kubenswrapper[4771]: I1011 10:28:30.246520 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-encryption-config\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.247163 master-1 kubenswrapper[4771]: I1011 10:28:30.247085 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-client\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.247352 master-1 kubenswrapper[4771]: I1011 10:28:30.247160 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-serving-cert\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.262058 master-1 kubenswrapper[4771]: I1011 10:28:30.261993 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkqb\" (UniqueName: \"kubernetes.io/projected/94d811f4-4ac9-46b0-b937-d3370b1b4305-kube-api-access-dbkqb\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.608307 master-1 kubenswrapper[4771]: I1011 10:28:30.608246 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-controller-manager/controller-manager-5d9b59775c-wqj5f"] Oct 11 10:28:30.609139 master-1 kubenswrapper[4771]: I1011 10:28:30.609106 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.612871 master-1 kubenswrapper[4771]: I1011 10:28:30.612814 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"config" Oct 11 10:28:30.613014 master-1 kubenswrapper[4771]: I1011 10:28:30.612882 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-controller-manager"/"serving-cert" Oct 11 10:28:30.613014 master-1 kubenswrapper[4771]: I1011 10:28:30.612893 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"client-ca" Oct 11 10:28:30.613297 master-1 kubenswrapper[4771]: I1011 10:28:30.613259 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-service-ca.crt" Oct 11 10:28:30.613413 master-1 kubenswrapper[4771]: I1011 10:28:30.613309 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"kube-root-ca.crt" Oct 11 10:28:30.614804 master-1 kubenswrapper[4771]: I1011 10:28:30.614772 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-global-ca" Oct 11 10:28:30.624149 master-1 kubenswrapper[4771]: I1011 10:28:30.624037 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-controller-manager/controller-manager-5d9b59775c-wqj5f"] Oct 11 10:28:30.644076 master-1 kubenswrapper[4771]: I1011 10:28:30.644024 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.644429 master-1 kubenswrapper[4771]: I1011 10:28:30.644405 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhr88\" (UniqueName: \"kubernetes.io/projected/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-kube-api-access-vhr88\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.644609 master-1 kubenswrapper[4771]: I1011 10:28:30.644586 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.644734 master-1 kubenswrapper[4771]: I1011 10:28:30.644717 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.644979 master-1 kubenswrapper[4771]: I1011 10:28:30.644962 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.746614 master-1 kubenswrapper[4771]: I1011 10:28:30.746307 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.746895 master-1 kubenswrapper[4771]: I1011 10:28:30.746636 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-vhr88\" (UniqueName: \"kubernetes.io/projected/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-kube-api-access-vhr88\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.746895 master-1 kubenswrapper[4771]: I1011 10:28:30.746710 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.746895 master-1 kubenswrapper[4771]: I1011 10:28:30.746750 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.746895 master-1 kubenswrapper[4771]: I1011 10:28:30.746787 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:30.746895 master-1 kubenswrapper[4771]: I1011 10:28:30.746827 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:30.747233 master-1 kubenswrapper[4771]: E1011 10:28:30.746940 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:30.747233 master-1 kubenswrapper[4771]: E1011 10:28:30.746968 4771 configmap.go:193] Couldn't get configMap openshift-apiserver/audit-0: configmap "audit-0" not found Oct 11 10:28:30.747233 master-1 kubenswrapper[4771]: E1011 10:28:30.747007 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:31.24698636 +0000 UTC m=+143.221212821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : configmap "client-ca" not found Oct 11 10:28:30.747233 master-1 kubenswrapper[4771]: E1011 10:28:30.747016 4771 secret.go:189] Couldn't get secret openshift-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:30.747233 master-1 kubenswrapper[4771]: E1011 10:28:30.747042 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit podName:94d811f4-4ac9-46b0-b937-d3370b1b4305 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:31.747019301 +0000 UTC m=+143.721245782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "audit" (UniqueName: "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit") pod "apiserver-796c687c6d-9b677" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305") : configmap "audit-0" not found Oct 11 10:28:30.747233 master-1 kubenswrapper[4771]: E1011 10:28:30.747092 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:31.247065892 +0000 UTC m=+143.221292373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : secret "serving-cert" not found Oct 11 10:28:30.747233 master-1 kubenswrapper[4771]: E1011 10:28:30.747105 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/openshift-global-ca: configmap "openshift-global-ca" not found Oct 11 10:28:30.747233 master-1 kubenswrapper[4771]: E1011 10:28:30.747207 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/config: configmap "config" not found Oct 11 10:28:30.747233 master-1 kubenswrapper[4771]: E1011 10:28:30.747235 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:31.247216756 +0000 UTC m=+143.221443237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-ca-bundles" (UniqueName: "kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : configmap "openshift-global-ca" not found Oct 11 10:28:30.747893 master-1 kubenswrapper[4771]: E1011 10:28:30.747265 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:31.247248347 +0000 UTC m=+143.221474798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config" (UniqueName: "kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : configmap "config" not found Oct 11 10:28:30.780795 master-1 kubenswrapper[4771]: I1011 10:28:30.780679 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhr88\" (UniqueName: \"kubernetes.io/projected/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-kube-api-access-vhr88\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:31.251887 master-1 kubenswrapper[4771]: I1011 10:28:31.251730 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:31.251887 master-1 kubenswrapper[4771]: I1011 10:28:31.251815 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:31.251887 master-1 kubenswrapper[4771]: I1011 10:28:31.251883 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:31.252870 master-1 kubenswrapper[4771]: E1011 10:28:31.251913 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/openshift-global-ca: configmap "openshift-global-ca" not found Oct 11 10:28:31.252870 master-1 kubenswrapper[4771]: E1011 10:28:31.252023 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:32.251995591 +0000 UTC m=+144.226222072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-ca-bundles" (UniqueName: "kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : configmap "openshift-global-ca" not found Oct 11 10:28:31.252870 master-1 kubenswrapper[4771]: E1011 10:28:31.252035 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/config: configmap "config" not found Oct 11 10:28:31.252870 master-1 kubenswrapper[4771]: E1011 10:28:31.252113 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:32.252089673 +0000 UTC m=+144.226316144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "config" (UniqueName: "kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : configmap "config" not found Oct 11 10:28:31.252870 master-1 kubenswrapper[4771]: E1011 10:28:31.252207 4771 secret.go:189] Couldn't get secret openshift-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:31.252870 master-1 kubenswrapper[4771]: E1011 10:28:31.252244 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:32.252231357 +0000 UTC m=+144.226457838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : secret "serving-cert" not found Oct 11 10:28:31.252870 master-1 kubenswrapper[4771]: E1011 10:28:31.252289 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:31.252870 master-1 kubenswrapper[4771]: E1011 10:28:31.252326 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:32.252315379 +0000 UTC m=+144.226541850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : configmap "client-ca" not found Oct 11 10:28:31.252870 master-1 kubenswrapper[4771]: I1011 10:28:31.251932 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:31.693295 master-1 kubenswrapper[4771]: I1011 10:28:31.693216 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-5d9b59775c-wqj5f"] Oct 11 10:28:31.693685 master-1 kubenswrapper[4771]: E1011 10:28:31.693527 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca config proxy-ca-bundles serving-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" podUID="295ceec5-4761-4cb3-95a7-cfc5cb35f03e" Oct 11 10:28:31.708242 master-1 kubenswrapper[4771]: I1011 10:28:31.708160 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf"] Oct 11 10:28:31.708883 master-1 kubenswrapper[4771]: I1011 10:28:31.708840 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.713345 master-1 kubenswrapper[4771]: I1011 10:28:31.713287 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"openshift-service-ca.crt" Oct 11 10:28:31.713463 master-1 kubenswrapper[4771]: I1011 10:28:31.713340 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"config" Oct 11 10:28:31.713550 master-1 kubenswrapper[4771]: I1011 10:28:31.713303 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"client-ca" Oct 11 10:28:31.713618 master-1 kubenswrapper[4771]: I1011 10:28:31.713409 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-route-controller-manager"/"serving-cert" Oct 11 10:28:31.716060 master-1 kubenswrapper[4771]: I1011 10:28:31.715922 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"kube-root-ca.crt" Oct 11 10:28:31.718255 master-1 kubenswrapper[4771]: I1011 10:28:31.718209 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf"] Oct 11 10:28:31.758273 master-1 kubenswrapper[4771]: I1011 10:28:31.758155 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-config\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.758273 master-1 kubenswrapper[4771]: I1011 10:28:31.758240 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:31.758273 master-1 kubenswrapper[4771]: I1011 10:28:31.758273 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.758695 master-1 kubenswrapper[4771]: I1011 10:28:31.758306 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.758695 master-1 kubenswrapper[4771]: I1011 10:28:31.758339 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcj47\" (UniqueName: \"kubernetes.io/projected/67be9a32-17b4-480c-98ba-caf9841bef6b-kube-api-access-fcj47\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.758695 master-1 kubenswrapper[4771]: E1011 10:28:31.758520 4771 configmap.go:193] Couldn't get configMap openshift-apiserver/audit-0: configmap "audit-0" not found Oct 11 10:28:31.759303 master-1 kubenswrapper[4771]: E1011 10:28:31.759233 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit podName:94d811f4-4ac9-46b0-b937-d3370b1b4305 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:33.759182181 +0000 UTC m=+145.733408632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "audit" (UniqueName: "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit") pod "apiserver-796c687c6d-9b677" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305") : configmap "audit-0" not found Oct 11 10:28:31.860080 master-1 kubenswrapper[4771]: I1011 10:28:31.859935 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.860080 master-1 kubenswrapper[4771]: I1011 10:28:31.860035 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.860507 master-1 kubenswrapper[4771]: I1011 10:28:31.860096 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-fcj47\" (UniqueName: \"kubernetes.io/projected/67be9a32-17b4-480c-98ba-caf9841bef6b-kube-api-access-fcj47\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.860507 master-1 kubenswrapper[4771]: E1011 10:28:31.860172 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:31.860507 master-1 kubenswrapper[4771]: E1011 10:28:31.860334 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:32.360290134 +0000 UTC m=+144.334516585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : configmap "client-ca" not found Oct 11 10:28:31.860868 master-1 kubenswrapper[4771]: I1011 10:28:31.860204 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-config\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.861150 master-1 kubenswrapper[4771]: E1011 10:28:31.861104 4771 secret.go:189] Couldn't get secret openshift-route-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:31.861228 master-1 kubenswrapper[4771]: E1011 10:28:31.861165 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:32.361151357 +0000 UTC m=+144.335377818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : secret "serving-cert" not found Oct 11 10:28:31.861727 master-1 kubenswrapper[4771]: I1011 10:28:31.861673 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-config\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:31.882061 master-1 kubenswrapper[4771]: I1011 10:28:31.881983 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcj47\" (UniqueName: \"kubernetes.io/projected/67be9a32-17b4-480c-98ba-caf9841bef6b-kube-api-access-fcj47\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:32.038719 master-1 kubenswrapper[4771]: I1011 10:28:32.038539 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:32.047936 master-1 kubenswrapper[4771]: I1011 10:28:32.047865 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:32.165175 master-1 kubenswrapper[4771]: I1011 10:28:32.165054 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhr88\" (UniqueName: \"kubernetes.io/projected/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-kube-api-access-vhr88\") pod \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " Oct 11 10:28:32.174689 master-1 kubenswrapper[4771]: I1011 10:28:32.174611 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-kube-api-access-vhr88" (OuterVolumeSpecName: "kube-api-access-vhr88") pod "295ceec5-4761-4cb3-95a7-cfc5cb35f03e" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e"). InnerVolumeSpecName "kube-api-access-vhr88". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:28:32.266584 master-1 kubenswrapper[4771]: I1011 10:28:32.266436 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:32.266584 master-1 kubenswrapper[4771]: I1011 10:28:32.266556 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:32.266584 master-1 kubenswrapper[4771]: I1011 10:28:32.266601 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:32.267512 master-1 kubenswrapper[4771]: I1011 10:28:32.266631 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:32.267512 master-1 kubenswrapper[4771]: I1011 10:28:32.266676 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-vhr88\" (UniqueName: \"kubernetes.io/projected/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-kube-api-access-vhr88\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:32.267512 master-1 kubenswrapper[4771]: E1011 10:28:32.266710 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:32.267512 master-1 kubenswrapper[4771]: E1011 10:28:32.266810 4771 secret.go:189] Couldn't get secret openshift-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:32.267512 master-1 kubenswrapper[4771]: E1011 10:28:32.266852 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:34.266802242 +0000 UTC m=+146.241028723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : configmap "client-ca" not found Oct 11 10:28:32.267512 master-1 kubenswrapper[4771]: E1011 10:28:32.266897 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert podName:295ceec5-4761-4cb3-95a7-cfc5cb35f03e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:34.266872214 +0000 UTC m=+146.241098695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert") pod "controller-manager-5d9b59775c-wqj5f" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e") : secret "serving-cert" not found Oct 11 10:28:32.267848 master-1 kubenswrapper[4771]: I1011 10:28:32.267686 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:32.268911 master-1 kubenswrapper[4771]: I1011 10:28:32.268853 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles\") pod \"controller-manager-5d9b59775c-wqj5f\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:32.367748 master-1 kubenswrapper[4771]: I1011 10:28:32.367652 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles\") pod \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " Oct 11 10:28:32.367748 master-1 kubenswrapper[4771]: I1011 10:28:32.367736 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config\") pod \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\" (UID: \"295ceec5-4761-4cb3-95a7-cfc5cb35f03e\") " Oct 11 10:28:32.368177 master-1 kubenswrapper[4771]: I1011 10:28:32.367911 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:32.368177 master-1 kubenswrapper[4771]: I1011 10:28:32.367946 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:32.368177 master-1 kubenswrapper[4771]: E1011 10:28:32.368087 4771 secret.go:189] Couldn't get secret openshift-route-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:32.368177 master-1 kubenswrapper[4771]: E1011 10:28:32.368147 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:33.368126861 +0000 UTC m=+145.342353302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : secret "serving-cert" not found Oct 11 10:28:32.368550 master-1 kubenswrapper[4771]: E1011 10:28:32.368337 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:32.368550 master-1 kubenswrapper[4771]: E1011 10:28:32.368495 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:33.36846196 +0000 UTC m=+145.342688431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : configmap "client-ca" not found Oct 11 10:28:32.368707 master-1 kubenswrapper[4771]: I1011 10:28:32.368597 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config" (OuterVolumeSpecName: "config") pod "295ceec5-4761-4cb3-95a7-cfc5cb35f03e" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:28:32.368993 master-1 kubenswrapper[4771]: I1011 10:28:32.368895 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles" (OuterVolumeSpecName: "proxy-ca-bundles") pod "295ceec5-4761-4cb3-95a7-cfc5cb35f03e" (UID: "295ceec5-4761-4cb3-95a7-cfc5cb35f03e"). InnerVolumeSpecName "proxy-ca-bundles". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:28:32.468600 master-1 kubenswrapper[4771]: I1011 10:28:32.468547 4771 reconciler_common.go:293] "Volume detached for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-proxy-ca-bundles\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:32.468600 master-1 kubenswrapper[4771]: I1011 10:28:32.468585 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:32.671323 master-1 kubenswrapper[4771]: I1011 10:28:32.670989 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:28:32.675858 master-1 kubenswrapper[4771]: I1011 10:28:32.675793 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-diagnostics"/"kube-root-ca.crt" Oct 11 10:28:32.686520 master-1 kubenswrapper[4771]: I1011 10:28:32.686282 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-diagnostics"/"openshift-service-ca.crt" Oct 11 10:28:32.698699 master-1 kubenswrapper[4771]: I1011 10:28:32.698617 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktrh\" (UniqueName: \"kubernetes.io/projected/0bde275d-f0a5-4bea-93f7-edd2077e46b4-kube-api-access-hktrh\") pod \"network-check-target-4pm7x\" (UID: \"0bde275d-f0a5-4bea-93f7-edd2077e46b4\") " pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:28:32.878746 master-1 kubenswrapper[4771]: I1011 10:28:32.878602 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:28:33.042127 master-1 kubenswrapper[4771]: I1011 10:28:33.042048 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-5d9b59775c-wqj5f" Oct 11 10:28:33.080327 master-1 kubenswrapper[4771]: I1011 10:28:33.080281 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-controller-manager/controller-manager-857df878cf-tz7h4"] Oct 11 10:28:33.080824 master-1 kubenswrapper[4771]: I1011 10:28:33.080779 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.082474 master-1 kubenswrapper[4771]: I1011 10:28:33.082403 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-5d9b59775c-wqj5f"] Oct 11 10:28:33.084953 master-1 kubenswrapper[4771]: I1011 10:28:33.084921 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-controller-manager"/"serving-cert" Oct 11 10:28:33.085115 master-1 kubenswrapper[4771]: I1011 10:28:33.084978 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"kube-root-ca.crt" Oct 11 10:28:33.085656 master-1 kubenswrapper[4771]: I1011 10:28:33.085272 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-controller-manager/controller-manager-5d9b59775c-wqj5f"] Oct 11 10:28:33.085758 master-1 kubenswrapper[4771]: I1011 10:28:33.085711 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"config" Oct 11 10:28:33.086748 master-1 kubenswrapper[4771]: I1011 10:28:33.086698 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-service-ca.crt" Oct 11 10:28:33.087320 master-1 kubenswrapper[4771]: I1011 10:28:33.087257 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"client-ca" Oct 11 10:28:33.091658 master-1 kubenswrapper[4771]: I1011 10:28:33.091612 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-controller-manager/controller-manager-857df878cf-tz7h4"] Oct 11 10:28:33.095938 master-1 kubenswrapper[4771]: I1011 10:28:33.095859 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-global-ca" Oct 11 10:28:33.098592 master-1 kubenswrapper[4771]: I1011 10:28:33.098238 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4pm7x"] Oct 11 10:28:33.108424 master-1 kubenswrapper[4771]: W1011 10:28:33.108300 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bde275d_f0a5_4bea_93f7_edd2077e46b4.slice/crio-edbdb16084802a4579f0e809d7a0a2e2997637364095207122799afce742249b WatchSource:0}: Error finding container edbdb16084802a4579f0e809d7a0a2e2997637364095207122799afce742249b: Status 404 returned error can't find the container with id edbdb16084802a4579f0e809d7a0a2e2997637364095207122799afce742249b Oct 11 10:28:33.177195 master-1 kubenswrapper[4771]: I1011 10:28:33.177131 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-config\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.177305 master-1 kubenswrapper[4771]: I1011 10:28:33.177213 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.177305 master-1 kubenswrapper[4771]: I1011 10:28:33.177258 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-proxy-ca-bundles\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.177442 master-1 kubenswrapper[4771]: I1011 10:28:33.177325 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktq9\" (UniqueName: \"kubernetes.io/projected/9deef4a8-bf40-4a1f-bd3f-764b298245b2-kube-api-access-hktq9\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.177442 master-1 kubenswrapper[4771]: I1011 10:28:33.177385 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.177442 master-1 kubenswrapper[4771]: I1011 10:28:33.177435 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:33.177547 master-1 kubenswrapper[4771]: I1011 10:28:33.177457 4771 reconciler_common.go:293] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/295ceec5-4761-4cb3-95a7-cfc5cb35f03e-client-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:33.279032 master-1 kubenswrapper[4771]: I1011 10:28:33.278650 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hktq9\" (UniqueName: \"kubernetes.io/projected/9deef4a8-bf40-4a1f-bd3f-764b298245b2-kube-api-access-hktq9\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.279032 master-1 kubenswrapper[4771]: I1011 10:28:33.279001 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.280293 master-1 kubenswrapper[4771]: I1011 10:28:33.279071 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-config\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.280293 master-1 kubenswrapper[4771]: I1011 10:28:33.279115 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.280293 master-1 kubenswrapper[4771]: I1011 10:28:33.279149 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-proxy-ca-bundles\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.280293 master-1 kubenswrapper[4771]: E1011 10:28:33.279340 4771 secret.go:189] Couldn't get secret openshift-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:33.280293 master-1 kubenswrapper[4771]: E1011 10:28:33.279422 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:33.280293 master-1 kubenswrapper[4771]: E1011 10:28:33.279467 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:33.779439263 +0000 UTC m=+145.753665744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : secret "serving-cert" not found Oct 11 10:28:33.280293 master-1 kubenswrapper[4771]: E1011 10:28:33.279684 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:33.779649869 +0000 UTC m=+145.753876350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : configmap "client-ca" not found Oct 11 10:28:33.281320 master-1 kubenswrapper[4771]: I1011 10:28:33.281236 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-config\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.281762 master-1 kubenswrapper[4771]: I1011 10:28:33.281700 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-proxy-ca-bundles\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.307454 master-1 kubenswrapper[4771]: I1011 10:28:33.307336 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktq9\" (UniqueName: \"kubernetes.io/projected/9deef4a8-bf40-4a1f-bd3f-764b298245b2-kube-api-access-hktq9\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.380333 master-1 kubenswrapper[4771]: I1011 10:28:33.380080 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:33.380333 master-1 kubenswrapper[4771]: I1011 10:28:33.380188 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:33.380333 master-1 kubenswrapper[4771]: E1011 10:28:33.380325 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:33.381037 master-1 kubenswrapper[4771]: E1011 10:28:33.380483 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:35.380451574 +0000 UTC m=+147.354678055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : configmap "client-ca" not found Oct 11 10:28:33.381037 master-1 kubenswrapper[4771]: E1011 10:28:33.380334 4771 secret.go:189] Couldn't get secret openshift-route-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:33.381037 master-1 kubenswrapper[4771]: E1011 10:28:33.380623 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:35.380590328 +0000 UTC m=+147.354816869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : secret "serving-cert" not found Oct 11 10:28:33.785112 master-1 kubenswrapper[4771]: I1011 10:28:33.785045 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.785387 master-1 kubenswrapper[4771]: I1011 10:28:33.785179 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:33.785387 master-1 kubenswrapper[4771]: I1011 10:28:33.785244 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:33.785387 master-1 kubenswrapper[4771]: E1011 10:28:33.785312 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:33.785520 master-1 kubenswrapper[4771]: E1011 10:28:33.785411 4771 configmap.go:193] Couldn't get configMap openshift-apiserver/audit-0: configmap "audit-0" not found Oct 11 10:28:33.785520 master-1 kubenswrapper[4771]: E1011 10:28:33.785313 4771 secret.go:189] Couldn't get secret openshift-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:33.785596 master-1 kubenswrapper[4771]: E1011 10:28:33.785436 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:34.785413561 +0000 UTC m=+146.759640012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : configmap "client-ca" not found Oct 11 10:28:33.785596 master-1 kubenswrapper[4771]: E1011 10:28:33.785581 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit podName:94d811f4-4ac9-46b0-b937-d3370b1b4305 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:37.785548144 +0000 UTC m=+149.759774625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "audit" (UniqueName: "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit") pod "apiserver-796c687c6d-9b677" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305") : configmap "audit-0" not found Oct 11 10:28:33.785690 master-1 kubenswrapper[4771]: E1011 10:28:33.785619 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:34.785605896 +0000 UTC m=+146.759832397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : secret "serving-cert" not found Oct 11 10:28:34.047467 master-1 kubenswrapper[4771]: I1011 10:28:34.047232 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4pm7x" event={"ID":"0bde275d-f0a5-4bea-93f7-edd2077e46b4","Type":"ContainerStarted","Data":"88e945236d6876de3adbd520e22a037ce30e88ce8cfbf1da3226eb75f03ff32f"} Oct 11 10:28:34.047467 master-1 kubenswrapper[4771]: I1011 10:28:34.047317 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4pm7x" event={"ID":"0bde275d-f0a5-4bea-93f7-edd2077e46b4","Type":"ContainerStarted","Data":"edbdb16084802a4579f0e809d7a0a2e2997637364095207122799afce742249b"} Oct 11 10:28:34.047467 master-1 kubenswrapper[4771]: I1011 10:28:34.047429 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:28:34.063905 master-1 kubenswrapper[4771]: I1011 10:28:34.063787 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4pm7x" podStartSLOduration=66.063763757 podStartE2EDuration="1m6.063763757s" podCreationTimestamp="2025-10-11 10:27:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:28:34.062802522 +0000 UTC m=+146.037028973" watchObservedRunningTime="2025-10-11 10:28:34.063763757 +0000 UTC m=+146.037990238" Oct 11 10:28:34.437804 master-1 kubenswrapper[4771]: I1011 10:28:34.437702 4771 scope.go:117] "RemoveContainer" containerID="793c72629ffb5d64763cce906980f11774530f02d707e0389b69155b33560c5d" Oct 11 10:28:34.438662 master-1 kubenswrapper[4771]: E1011 10:28:34.437953 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-779749f859-5xxzp_openshift-cloud-controller-manager-operator(e115f8be-9e65-4407-8111-568e5ea8ac1b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" podUID="e115f8be-9e65-4407-8111-568e5ea8ac1b" Oct 11 10:28:34.450790 master-1 kubenswrapper[4771]: I1011 10:28:34.450723 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="295ceec5-4761-4cb3-95a7-cfc5cb35f03e" path="/var/lib/kubelet/pods/295ceec5-4761-4cb3-95a7-cfc5cb35f03e/volumes" Oct 11 10:28:34.795475 master-1 kubenswrapper[4771]: I1011 10:28:34.795270 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:34.795475 master-1 kubenswrapper[4771]: I1011 10:28:34.795459 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:34.795766 master-1 kubenswrapper[4771]: E1011 10:28:34.795622 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:34.795766 master-1 kubenswrapper[4771]: E1011 10:28:34.795697 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:36.795675961 +0000 UTC m=+148.769902442 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : configmap "client-ca" not found Oct 11 10:28:34.796151 master-1 kubenswrapper[4771]: E1011 10:28:34.796042 4771 secret.go:189] Couldn't get secret openshift-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:34.796151 master-1 kubenswrapper[4771]: E1011 10:28:34.796124 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:36.796105552 +0000 UTC m=+148.770332033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : secret "serving-cert" not found Oct 11 10:28:35.401387 master-1 kubenswrapper[4771]: I1011 10:28:35.401259 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:35.401387 master-1 kubenswrapper[4771]: I1011 10:28:35.401339 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:35.401960 master-1 kubenswrapper[4771]: E1011 10:28:35.401497 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:35.401960 master-1 kubenswrapper[4771]: E1011 10:28:35.401518 4771 secret.go:189] Couldn't get secret openshift-route-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:35.401960 master-1 kubenswrapper[4771]: E1011 10:28:35.401608 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:39.40158344 +0000 UTC m=+151.375809911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : secret "serving-cert" not found Oct 11 10:28:35.401960 master-1 kubenswrapper[4771]: E1011 10:28:35.401633 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:39.401621951 +0000 UTC m=+151.375848432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : configmap "client-ca" not found Oct 11 10:28:36.817819 master-1 kubenswrapper[4771]: I1011 10:28:36.817719 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:36.818700 master-1 kubenswrapper[4771]: E1011 10:28:36.817840 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:36.818700 master-1 kubenswrapper[4771]: I1011 10:28:36.817889 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:36.818700 master-1 kubenswrapper[4771]: E1011 10:28:36.817920 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:40.817899774 +0000 UTC m=+152.792126235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : configmap "client-ca" not found Oct 11 10:28:36.818700 master-1 kubenswrapper[4771]: E1011 10:28:36.818117 4771 secret.go:189] Couldn't get secret openshift-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:36.818700 master-1 kubenswrapper[4771]: E1011 10:28:36.818208 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:40.818182442 +0000 UTC m=+152.792408923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : secret "serving-cert" not found Oct 11 10:28:37.829270 master-1 kubenswrapper[4771]: I1011 10:28:37.829146 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:37.830065 master-1 kubenswrapper[4771]: E1011 10:28:37.829320 4771 configmap.go:193] Couldn't get configMap openshift-apiserver/audit-0: configmap "audit-0" not found Oct 11 10:28:37.830065 master-1 kubenswrapper[4771]: E1011 10:28:37.829467 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit podName:94d811f4-4ac9-46b0-b937-d3370b1b4305 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:45.829441428 +0000 UTC m=+157.803667919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "audit" (UniqueName: "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit") pod "apiserver-796c687c6d-9b677" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305") : configmap "audit-0" not found Oct 11 10:28:39.447972 master-1 kubenswrapper[4771]: I1011 10:28:39.447689 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:39.447972 master-1 kubenswrapper[4771]: I1011 10:28:39.447752 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:39.447972 master-1 kubenswrapper[4771]: E1011 10:28:39.447864 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:39.449047 master-1 kubenswrapper[4771]: E1011 10:28:39.447996 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:47.447966079 +0000 UTC m=+159.422192550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : configmap "client-ca" not found Oct 11 10:28:39.449047 master-1 kubenswrapper[4771]: E1011 10:28:39.447886 4771 secret.go:189] Couldn't get secret openshift-route-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:39.449047 master-1 kubenswrapper[4771]: E1011 10:28:39.448070 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:47.448053542 +0000 UTC m=+159.422280063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : secret "serving-cert" not found Oct 11 10:28:40.493556 master-1 kubenswrapper[4771]: I1011 10:28:40.493463 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/installer-1-master-1"] Oct 11 10:28:40.494386 master-1 kubenswrapper[4771]: I1011 10:28:40.493971 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.497645 master-1 kubenswrapper[4771]: I1011 10:28:40.497588 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-scheduler"/"kube-root-ca.crt" Oct 11 10:28:40.548816 master-1 kubenswrapper[4771]: I1011 10:28:40.501772 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-1-master-1"] Oct 11 10:28:40.560863 master-1 kubenswrapper[4771]: I1011 10:28:40.560770 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kube-api-access\") pod \"installer-1-master-1\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.560863 master-1 kubenswrapper[4771]: I1011 10:28:40.560859 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.561205 master-1 kubenswrapper[4771]: I1011 10:28:40.561102 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-var-lock\") pod \"installer-1-master-1\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.662548 master-1 kubenswrapper[4771]: I1011 10:28:40.662390 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-var-lock\") pod \"installer-1-master-1\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.662548 master-1 kubenswrapper[4771]: I1011 10:28:40.662554 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kube-api-access\") pod \"installer-1-master-1\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.662957 master-1 kubenswrapper[4771]: I1011 10:28:40.662554 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-var-lock\") pod \"installer-1-master-1\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.662957 master-1 kubenswrapper[4771]: I1011 10:28:40.662681 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.662957 master-1 kubenswrapper[4771]: I1011 10:28:40.662609 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.687191 master-1 kubenswrapper[4771]: I1011 10:28:40.687093 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kube-api-access\") pod \"installer-1-master-1\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.856945 master-1 kubenswrapper[4771]: I1011 10:28:40.856833 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:28:40.865451 master-1 kubenswrapper[4771]: I1011 10:28:40.865395 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:40.865557 master-1 kubenswrapper[4771]: I1011 10:28:40.865486 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:40.865706 master-1 kubenswrapper[4771]: E1011 10:28:40.865663 4771 secret.go:189] Couldn't get secret openshift-controller-manager/serving-cert: secret "serving-cert" not found Oct 11 10:28:40.865781 master-1 kubenswrapper[4771]: E1011 10:28:40.865723 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:40.865854 master-1 kubenswrapper[4771]: E1011 10:28:40.865750 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:48.865729562 +0000 UTC m=+160.839956013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : secret "serving-cert" not found Oct 11 10:28:40.865933 master-1 kubenswrapper[4771]: E1011 10:28:40.865863 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:48.865837655 +0000 UTC m=+160.840064196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : configmap "client-ca" not found Oct 11 10:28:41.096526 master-1 kubenswrapper[4771]: I1011 10:28:41.096340 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-1-master-1"] Oct 11 10:28:41.106135 master-1 kubenswrapper[4771]: W1011 10:28:41.105986 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod007dcbab_9e3e_4dcd_9ad9_0ea8dd07dfc7.slice/crio-862c7a0762d99806cbf395198a8a115efc48c49b1a93e1cd22e9f82545990f2e WatchSource:0}: Error finding container 862c7a0762d99806cbf395198a8a115efc48c49b1a93e1cd22e9f82545990f2e: Status 404 returned error can't find the container with id 862c7a0762d99806cbf395198a8a115efc48c49b1a93e1cd22e9f82545990f2e Oct 11 10:28:42.072232 master-1 kubenswrapper[4771]: I1011 10:28:42.071976 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-1-master-1" event={"ID":"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7","Type":"ContainerStarted","Data":"862c7a0762d99806cbf395198a8a115efc48c49b1a93e1cd22e9f82545990f2e"} Oct 11 10:28:43.418786 master-1 kubenswrapper[4771]: I1011 10:28:43.418599 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw"] Oct 11 10:28:43.419856 master-1 kubenswrapper[4771]: I1011 10:28:43.419246 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.423335 master-1 kubenswrapper[4771]: I1011 10:28:43.423264 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"audit-1" Oct 11 10:28:43.426139 master-1 kubenswrapper[4771]: I1011 10:28:43.426051 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"etcd-client" Oct 11 10:28:43.426436 master-1 kubenswrapper[4771]: I1011 10:28:43.426208 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"encryption-config-1" Oct 11 10:28:43.426555 master-1 kubenswrapper[4771]: I1011 10:28:43.426457 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"kube-root-ca.crt" Oct 11 10:28:43.426555 master-1 kubenswrapper[4771]: I1011 10:28:43.426461 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"openshift-service-ca.crt" Oct 11 10:28:43.426723 master-1 kubenswrapper[4771]: I1011 10:28:43.426596 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"etcd-serving-ca" Oct 11 10:28:43.426936 master-1 kubenswrapper[4771]: I1011 10:28:43.426863 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"serving-cert" Oct 11 10:28:43.427041 master-1 kubenswrapper[4771]: I1011 10:28:43.426876 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"trusted-ca-bundle" Oct 11 10:28:43.432081 master-1 kubenswrapper[4771]: I1011 10:28:43.432015 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw"] Oct 11 10:28:43.493571 master-1 kubenswrapper[4771]: I1011 10:28:43.493496 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-client\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.493824 master-1 kubenswrapper[4771]: I1011 10:28:43.493587 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-serving-ca\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.493824 master-1 kubenswrapper[4771]: I1011 10:28:43.493625 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-dir\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.493824 master-1 kubenswrapper[4771]: I1011 10:28:43.493750 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrj8w\" (UniqueName: \"kubernetes.io/projected/004ee387-d0e9-4582-ad14-f571832ebd6e-kube-api-access-vrj8w\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.494008 master-1 kubenswrapper[4771]: I1011 10:28:43.493838 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.494008 master-1 kubenswrapper[4771]: I1011 10:28:43.493975 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-trusted-ca-bundle\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.494131 master-1 kubenswrapper[4771]: I1011 10:28:43.494068 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-policies\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.494131 master-1 kubenswrapper[4771]: I1011 10:28:43.494105 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-encryption-config\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.595386 master-1 kubenswrapper[4771]: I1011 10:28:43.595230 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-trusted-ca-bundle\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.595386 master-1 kubenswrapper[4771]: I1011 10:28:43.595345 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-policies\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.595710 master-1 kubenswrapper[4771]: I1011 10:28:43.595413 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-encryption-config\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.595710 master-1 kubenswrapper[4771]: I1011 10:28:43.595457 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-client\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.595710 master-1 kubenswrapper[4771]: I1011 10:28:43.595488 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-serving-ca\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.595710 master-1 kubenswrapper[4771]: I1011 10:28:43.595521 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-dir\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.595710 master-1 kubenswrapper[4771]: I1011 10:28:43.595568 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-vrj8w\" (UniqueName: \"kubernetes.io/projected/004ee387-d0e9-4582-ad14-f571832ebd6e-kube-api-access-vrj8w\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.595710 master-1 kubenswrapper[4771]: I1011 10:28:43.595615 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.596551 master-1 kubenswrapper[4771]: E1011 10:28:43.595786 4771 secret.go:189] Couldn't get secret openshift-oauth-apiserver/serving-cert: secret "serving-cert" not found Oct 11 10:28:43.596551 master-1 kubenswrapper[4771]: E1011 10:28:43.595861 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert podName:004ee387-d0e9-4582-ad14-f571832ebd6e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:44.0958376 +0000 UTC m=+156.070064051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert") pod "apiserver-65b6f4d4c9-skwvw" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e") : secret "serving-cert" not found Oct 11 10:28:43.596551 master-1 kubenswrapper[4771]: I1011 10:28:43.595917 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-dir\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.596551 master-1 kubenswrapper[4771]: I1011 10:28:43.596455 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-trusted-ca-bundle\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.596932 master-1 kubenswrapper[4771]: I1011 10:28:43.596861 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-policies\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.597103 master-1 kubenswrapper[4771]: I1011 10:28:43.597054 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-serving-ca\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.603835 master-1 kubenswrapper[4771]: I1011 10:28:43.603776 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-encryption-config\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.603995 master-1 kubenswrapper[4771]: I1011 10:28:43.603928 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-client\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:43.625311 master-1 kubenswrapper[4771]: I1011 10:28:43.625218 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrj8w\" (UniqueName: \"kubernetes.io/projected/004ee387-d0e9-4582-ad14-f571832ebd6e-kube-api-access-vrj8w\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:44.081613 master-1 kubenswrapper[4771]: I1011 10:28:44.081481 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-1-master-1" event={"ID":"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7","Type":"ContainerStarted","Data":"80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90"} Oct 11 10:28:44.100508 master-1 kubenswrapper[4771]: I1011 10:28:44.100424 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:44.100791 master-1 kubenswrapper[4771]: E1011 10:28:44.100641 4771 secret.go:189] Couldn't get secret openshift-oauth-apiserver/serving-cert: secret "serving-cert" not found Oct 11 10:28:44.100791 master-1 kubenswrapper[4771]: E1011 10:28:44.100759 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert podName:004ee387-d0e9-4582-ad14-f571832ebd6e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:45.100731779 +0000 UTC m=+157.074958260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert") pod "apiserver-65b6f4d4c9-skwvw" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e") : secret "serving-cert" not found Oct 11 10:28:44.103228 master-1 kubenswrapper[4771]: I1011 10:28:44.103139 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/installer-1-master-1" podStartSLOduration=2.097822787 podStartE2EDuration="4.103118512s" podCreationTimestamp="2025-10-11 10:28:40 +0000 UTC" firstStartedPulling="2025-10-11 10:28:41.109361787 +0000 UTC m=+153.083588238" lastFinishedPulling="2025-10-11 10:28:43.114657522 +0000 UTC m=+155.088883963" observedRunningTime="2025-10-11 10:28:44.100324648 +0000 UTC m=+156.074551119" watchObservedRunningTime="2025-10-11 10:28:44.103118512 +0000 UTC m=+156.077344983" Oct 11 10:28:45.112290 master-1 kubenswrapper[4771]: I1011 10:28:45.112182 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:45.113337 master-1 kubenswrapper[4771]: E1011 10:28:45.112473 4771 secret.go:189] Couldn't get secret openshift-oauth-apiserver/serving-cert: secret "serving-cert" not found Oct 11 10:28:45.113337 master-1 kubenswrapper[4771]: E1011 10:28:45.112599 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert podName:004ee387-d0e9-4582-ad14-f571832ebd6e nodeName:}" failed. No retries permitted until 2025-10-11 10:28:47.11256841 +0000 UTC m=+159.086794891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert") pod "apiserver-65b6f4d4c9-skwvw" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e") : secret "serving-cert" not found Oct 11 10:28:45.244382 master-1 kubenswrapper[4771]: I1011 10:28:45.244291 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc"] Oct 11 10:28:45.245147 master-1 kubenswrapper[4771]: I1011 10:28:45.245079 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.247998 master-1 kubenswrapper[4771]: I1011 10:28:45.247944 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-operator-controller"/"kube-root-ca.crt" Oct 11 10:28:45.248594 master-1 kubenswrapper[4771]: I1011 10:28:45.248542 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-operator-controller"/"openshift-service-ca.crt" Oct 11 10:28:45.252669 master-1 kubenswrapper[4771]: I1011 10:28:45.252624 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc"] Oct 11 10:28:45.257644 master-1 kubenswrapper[4771]: I1011 10:28:45.257587 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-operator-controller"/"operator-controller-trusted-ca-bundle" Oct 11 10:28:45.314890 master-1 kubenswrapper[4771]: I1011 10:28:45.314840 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.315090 master-1 kubenswrapper[4771]: I1011 10:28:45.314907 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cache\" (UniqueName: \"kubernetes.io/empty-dir/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-cache\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.315090 master-1 kubenswrapper[4771]: I1011 10:28:45.314997 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-containers\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-containers\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.315224 master-1 kubenswrapper[4771]: I1011 10:28:45.315158 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-certs\" (UniqueName: \"kubernetes.io/projected/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-ca-certs\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.315288 master-1 kubenswrapper[4771]: I1011 10:28:45.315248 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbvx\" (UniqueName: \"kubernetes.io/projected/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-kube-api-access-crbvx\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.340613 master-1 kubenswrapper[4771]: I1011 10:28:45.340542 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm"] Oct 11 10:28:45.341478 master-1 kubenswrapper[4771]: I1011 10:28:45.341430 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.347463 master-1 kubenswrapper[4771]: I1011 10:28:45.347403 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-catalogd"/"openshift-service-ca.crt" Oct 11 10:28:45.347624 master-1 kubenswrapper[4771]: I1011 10:28:45.347446 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-catalogd"/"kube-root-ca.crt" Oct 11 10:28:45.347624 master-1 kubenswrapper[4771]: I1011 10:28:45.347527 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-catalogd"/"catalogserver-cert" Oct 11 10:28:45.352565 master-1 kubenswrapper[4771]: I1011 10:28:45.352522 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm"] Oct 11 10:28:45.357694 master-1 kubenswrapper[4771]: I1011 10:28:45.357627 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-catalogd"/"catalogd-trusted-ca-bundle" Oct 11 10:28:45.416215 master-1 kubenswrapper[4771]: I1011 10:28:45.416009 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalogserver-certs\" (UniqueName: \"kubernetes.io/secret/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-catalogserver-certs\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.416215 master-1 kubenswrapper[4771]: I1011 10:28:45.416106 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-certs\" (UniqueName: \"kubernetes.io/projected/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-ca-certs\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.416699 master-1 kubenswrapper[4771]: I1011 10:28:45.416216 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.416699 master-1 kubenswrapper[4771]: I1011 10:28:45.416282 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qknv\" (UniqueName: \"kubernetes.io/projected/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-kube-api-access-9qknv\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.416699 master-1 kubenswrapper[4771]: I1011 10:28:45.416345 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cache\" (UniqueName: \"kubernetes.io/empty-dir/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-cache\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.416699 master-1 kubenswrapper[4771]: I1011 10:28:45.416436 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-containers\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-containers\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.416699 master-1 kubenswrapper[4771]: I1011 10:28:45.416512 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cache\" (UniqueName: \"kubernetes.io/empty-dir/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-cache\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.416699 master-1 kubenswrapper[4771]: I1011 10:28:45.416586 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-containers\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-containers\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.416699 master-1 kubenswrapper[4771]: I1011 10:28:45.416675 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.417221 master-1 kubenswrapper[4771]: I1011 10:28:45.416778 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ca-certs\" (UniqueName: \"kubernetes.io/projected/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-ca-certs\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.417221 master-1 kubenswrapper[4771]: I1011 10:28:45.416861 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-crbvx\" (UniqueName: \"kubernetes.io/projected/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-kube-api-access-crbvx\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.417904 master-1 kubenswrapper[4771]: I1011 10:28:45.417852 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-containers\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-containers\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.418062 master-1 kubenswrapper[4771]: E1011 10:28:45.418007 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:28:45.917968284 +0000 UTC m=+157.892194765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:45.418797 master-1 kubenswrapper[4771]: I1011 10:28:45.418720 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cache\" (UniqueName: \"kubernetes.io/empty-dir/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-cache\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.423553 master-1 kubenswrapper[4771]: I1011 10:28:45.423474 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ca-certs\" (UniqueName: \"kubernetes.io/projected/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-ca-certs\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.436209 master-1 kubenswrapper[4771]: I1011 10:28:45.436114 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbvx\" (UniqueName: \"kubernetes.io/projected/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-kube-api-access-crbvx\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.518021 master-1 kubenswrapper[4771]: I1011 10:28:45.517853 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalogserver-certs\" (UniqueName: \"kubernetes.io/secret/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-catalogserver-certs\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.518021 master-1 kubenswrapper[4771]: I1011 10:28:45.517933 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ca-certs\" (UniqueName: \"kubernetes.io/projected/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-ca-certs\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.518021 master-1 kubenswrapper[4771]: I1011 10:28:45.517977 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-9qknv\" (UniqueName: \"kubernetes.io/projected/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-kube-api-access-9qknv\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.518607 master-1 kubenswrapper[4771]: I1011 10:28:45.518053 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cache\" (UniqueName: \"kubernetes.io/empty-dir/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-cache\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.518607 master-1 kubenswrapper[4771]: I1011 10:28:45.518129 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-containers\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-containers\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.518607 master-1 kubenswrapper[4771]: E1011 10:28:45.518141 4771 secret.go:189] Couldn't get secret openshift-catalogd/catalogserver-cert: secret "catalogserver-cert" not found Oct 11 10:28:45.518607 master-1 kubenswrapper[4771]: I1011 10:28:45.518217 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.518607 master-1 kubenswrapper[4771]: E1011 10:28:45.518269 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-catalogserver-certs podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:46.018232225 +0000 UTC m=+157.992458756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "catalogserver-certs" (UniqueName: "kubernetes.io/secret/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-catalogserver-certs") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : secret "catalogserver-cert" not found Oct 11 10:28:45.518607 master-1 kubenswrapper[4771]: I1011 10:28:45.518393 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-containers\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-containers\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.518607 master-1 kubenswrapper[4771]: E1011 10:28:45.518425 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:46.018398129 +0000 UTC m=+157.992624610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:45.519215 master-1 kubenswrapper[4771]: I1011 10:28:45.519097 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cache\" (UniqueName: \"kubernetes.io/empty-dir/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-cache\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.523590 master-1 kubenswrapper[4771]: I1011 10:28:45.523276 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ca-certs\" (UniqueName: \"kubernetes.io/projected/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-ca-certs\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.557169 master-1 kubenswrapper[4771]: I1011 10:28:45.557025 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qknv\" (UniqueName: \"kubernetes.io/projected/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-kube-api-access-9qknv\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:45.924576 master-1 kubenswrapper[4771]: I1011 10:28:45.924440 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit\") pod \"apiserver-796c687c6d-9b677\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:45.924870 master-1 kubenswrapper[4771]: E1011 10:28:45.924590 4771 configmap.go:193] Couldn't get configMap openshift-apiserver/audit-0: configmap "audit-0" not found Oct 11 10:28:45.924870 master-1 kubenswrapper[4771]: E1011 10:28:45.924691 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit podName:94d811f4-4ac9-46b0-b937-d3370b1b4305 nodeName:}" failed. No retries permitted until 2025-10-11 10:29:01.924665021 +0000 UTC m=+173.898891492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "audit" (UniqueName: "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit") pod "apiserver-796c687c6d-9b677" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305") : configmap "audit-0" not found Oct 11 10:28:45.924870 master-1 kubenswrapper[4771]: I1011 10:28:45.924712 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:45.925245 master-1 kubenswrapper[4771]: E1011 10:28:45.925171 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:28:46.925114392 +0000 UTC m=+158.899340883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:46.025910 master-1 kubenswrapper[4771]: I1011 10:28:46.025797 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:46.025910 master-1 kubenswrapper[4771]: I1011 10:28:46.025905 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalogserver-certs\" (UniqueName: \"kubernetes.io/secret/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-catalogserver-certs\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:46.026412 master-1 kubenswrapper[4771]: E1011 10:28:46.026066 4771 secret.go:189] Couldn't get secret openshift-catalogd/catalogserver-cert: secret "catalogserver-cert" not found Oct 11 10:28:46.026412 master-1 kubenswrapper[4771]: E1011 10:28:46.026135 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:47.026094772 +0000 UTC m=+159.000321253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:46.026412 master-1 kubenswrapper[4771]: E1011 10:28:46.026190 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-catalogserver-certs podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:47.026174434 +0000 UTC m=+159.000400915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "catalogserver-certs" (UniqueName: "kubernetes.io/secret/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-catalogserver-certs") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : secret "catalogserver-cert" not found Oct 11 10:28:46.938605 master-1 kubenswrapper[4771]: I1011 10:28:46.938464 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:46.939185 master-1 kubenswrapper[4771]: E1011 10:28:46.938715 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:28:48.938681898 +0000 UTC m=+160.912908369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:47.039633 master-1 kubenswrapper[4771]: I1011 10:28:47.039504 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:47.039891 master-1 kubenswrapper[4771]: I1011 10:28:47.039652 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalogserver-certs\" (UniqueName: \"kubernetes.io/secret/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-catalogserver-certs\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:47.039948 master-1 kubenswrapper[4771]: E1011 10:28:47.039879 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:49.039837413 +0000 UTC m=+161.014063884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:47.044594 master-1 kubenswrapper[4771]: I1011 10:28:47.044543 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalogserver-certs\" (UniqueName: \"kubernetes.io/secret/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-catalogserver-certs\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:47.141320 master-1 kubenswrapper[4771]: I1011 10:28:47.140670 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:47.145458 master-1 kubenswrapper[4771]: I1011 10:28:47.145397 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert\") pod \"apiserver-65b6f4d4c9-skwvw\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:47.340547 master-1 kubenswrapper[4771]: I1011 10:28:47.340435 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:47.545573 master-1 kubenswrapper[4771]: I1011 10:28:47.545482 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:47.545823 master-1 kubenswrapper[4771]: I1011 10:28:47.545637 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:47.545823 master-1 kubenswrapper[4771]: E1011 10:28:47.545652 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:47.545823 master-1 kubenswrapper[4771]: E1011 10:28:47.545752 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca podName:67be9a32-17b4-480c-98ba-caf9841bef6b nodeName:}" failed. No retries permitted until 2025-10-11 10:29:03.545729278 +0000 UTC m=+175.519955769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca") pod "route-controller-manager-67d4d4d6d8-szbpf" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b") : configmap "client-ca" not found Oct 11 10:28:47.550486 master-1 kubenswrapper[4771]: I1011 10:28:47.550434 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") pod \"route-controller-manager-67d4d4d6d8-szbpf\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:47.556673 master-1 kubenswrapper[4771]: I1011 10:28:47.556596 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw"] Oct 11 10:28:47.566553 master-1 kubenswrapper[4771]: W1011 10:28:47.566488 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004ee387_d0e9_4582_ad14_f571832ebd6e.slice/crio-70ee09355a354a55a1e3cc86654a95e054448e4680cbf989813075d48bc93f03 WatchSource:0}: Error finding container 70ee09355a354a55a1e3cc86654a95e054448e4680cbf989813075d48bc93f03: Status 404 returned error can't find the container with id 70ee09355a354a55a1e3cc86654a95e054448e4680cbf989813075d48bc93f03 Oct 11 10:28:48.096924 master-1 kubenswrapper[4771]: I1011 10:28:48.096861 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" event={"ID":"004ee387-d0e9-4582-ad14-f571832ebd6e","Type":"ContainerStarted","Data":"70ee09355a354a55a1e3cc86654a95e054448e4680cbf989813075d48bc93f03"} Oct 11 10:28:48.483741 master-1 kubenswrapper[4771]: I1011 10:28:48.483628 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rzjcf"] Oct 11 10:28:48.484582 master-1 kubenswrapper[4771]: I1011 10:28:48.484555 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.487997 master-1 kubenswrapper[4771]: I1011 10:28:48.487950 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-dns"/"dns-default" Oct 11 10:28:48.488163 master-1 kubenswrapper[4771]: I1011 10:28:48.488110 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-dns"/"kube-root-ca.crt" Oct 11 10:28:48.488163 master-1 kubenswrapper[4771]: I1011 10:28:48.488160 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-dns"/"openshift-service-ca.crt" Oct 11 10:28:48.488302 master-1 kubenswrapper[4771]: I1011 10:28:48.488280 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-dns"/"dns-default-metrics-tls" Oct 11 10:28:48.493166 master-1 kubenswrapper[4771]: I1011 10:28:48.492874 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rzjcf"] Oct 11 10:28:48.555816 master-1 kubenswrapper[4771]: I1011 10:28:48.555725 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f49f37-a9e4-4acd-ae7e-d644e8475106-config-volume\") pod \"dns-default-rzjcf\" (UID: \"b3f49f37-a9e4-4acd-ae7e-d644e8475106\") " pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.555816 master-1 kubenswrapper[4771]: I1011 10:28:48.555786 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3f49f37-a9e4-4acd-ae7e-d644e8475106-metrics-tls\") pod \"dns-default-rzjcf\" (UID: \"b3f49f37-a9e4-4acd-ae7e-d644e8475106\") " pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.555816 master-1 kubenswrapper[4771]: I1011 10:28:48.555813 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzmn\" (UniqueName: \"kubernetes.io/projected/b3f49f37-a9e4-4acd-ae7e-d644e8475106-kube-api-access-xrzmn\") pod \"dns-default-rzjcf\" (UID: \"b3f49f37-a9e4-4acd-ae7e-d644e8475106\") " pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.657126 master-1 kubenswrapper[4771]: I1011 10:28:48.656498 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f49f37-a9e4-4acd-ae7e-d644e8475106-config-volume\") pod \"dns-default-rzjcf\" (UID: \"b3f49f37-a9e4-4acd-ae7e-d644e8475106\") " pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.657126 master-1 kubenswrapper[4771]: I1011 10:28:48.656574 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3f49f37-a9e4-4acd-ae7e-d644e8475106-metrics-tls\") pod \"dns-default-rzjcf\" (UID: \"b3f49f37-a9e4-4acd-ae7e-d644e8475106\") " pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.657126 master-1 kubenswrapper[4771]: I1011 10:28:48.656607 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzmn\" (UniqueName: \"kubernetes.io/projected/b3f49f37-a9e4-4acd-ae7e-d644e8475106-kube-api-access-xrzmn\") pod \"dns-default-rzjcf\" (UID: \"b3f49f37-a9e4-4acd-ae7e-d644e8475106\") " pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.657367 master-1 kubenswrapper[4771]: I1011 10:28:48.657256 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f49f37-a9e4-4acd-ae7e-d644e8475106-config-volume\") pod \"dns-default-rzjcf\" (UID: \"b3f49f37-a9e4-4acd-ae7e-d644e8475106\") " pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.659708 master-1 kubenswrapper[4771]: I1011 10:28:48.659693 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3f49f37-a9e4-4acd-ae7e-d644e8475106-metrics-tls\") pod \"dns-default-rzjcf\" (UID: \"b3f49f37-a9e4-4acd-ae7e-d644e8475106\") " pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.692504 master-1 kubenswrapper[4771]: I1011 10:28:48.692449 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzmn\" (UniqueName: \"kubernetes.io/projected/b3f49f37-a9e4-4acd-ae7e-d644e8475106-kube-api-access-xrzmn\") pod \"dns-default-rzjcf\" (UID: \"b3f49f37-a9e4-4acd-ae7e-d644e8475106\") " pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.782693 master-1 kubenswrapper[4771]: I1011 10:28:48.782542 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-fjwjw"] Oct 11 10:28:48.783170 master-1 kubenswrapper[4771]: I1011 10:28:48.783126 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fjwjw" Oct 11 10:28:48.805712 master-1 kubenswrapper[4771]: I1011 10:28:48.805659 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:48.859430 master-1 kubenswrapper[4771]: I1011 10:28:48.859377 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2919a957-a46f-4e96-b42e-3ba3c537e98e-hosts-file\") pod \"node-resolver-fjwjw\" (UID: \"2919a957-a46f-4e96-b42e-3ba3c537e98e\") " pod="openshift-dns/node-resolver-fjwjw" Oct 11 10:28:48.859641 master-1 kubenswrapper[4771]: I1011 10:28:48.859588 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpt6z\" (UniqueName: \"kubernetes.io/projected/2919a957-a46f-4e96-b42e-3ba3c537e98e-kube-api-access-zpt6z\") pod \"node-resolver-fjwjw\" (UID: \"2919a957-a46f-4e96-b42e-3ba3c537e98e\") " pod="openshift-dns/node-resolver-fjwjw" Oct 11 10:28:48.960499 master-1 kubenswrapper[4771]: I1011 10:28:48.960399 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:48.960499 master-1 kubenswrapper[4771]: I1011 10:28:48.960479 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-zpt6z\" (UniqueName: \"kubernetes.io/projected/2919a957-a46f-4e96-b42e-3ba3c537e98e-kube-api-access-zpt6z\") pod \"node-resolver-fjwjw\" (UID: \"2919a957-a46f-4e96-b42e-3ba3c537e98e\") " pod="openshift-dns/node-resolver-fjwjw" Oct 11 10:28:48.960865 master-1 kubenswrapper[4771]: I1011 10:28:48.960526 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:48.960865 master-1 kubenswrapper[4771]: I1011 10:28:48.960586 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:48.960865 master-1 kubenswrapper[4771]: I1011 10:28:48.960618 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2919a957-a46f-4e96-b42e-3ba3c537e98e-hosts-file\") pod \"node-resolver-fjwjw\" (UID: \"2919a957-a46f-4e96-b42e-3ba3c537e98e\") " pod="openshift-dns/node-resolver-fjwjw" Oct 11 10:28:48.960865 master-1 kubenswrapper[4771]: E1011 10:28:48.960723 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:48.960865 master-1 kubenswrapper[4771]: E1011 10:28:48.960778 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca podName:9deef4a8-bf40-4a1f-bd3f-764b298245b2 nodeName:}" failed. No retries permitted until 2025-10-11 10:29:04.960761319 +0000 UTC m=+176.934987770 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca") pod "controller-manager-857df878cf-tz7h4" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2") : configmap "client-ca" not found Oct 11 10:28:48.960865 master-1 kubenswrapper[4771]: E1011 10:28:48.960796 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:28:52.96078712 +0000 UTC m=+164.935013571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:48.960865 master-1 kubenswrapper[4771]: I1011 10:28:48.960731 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2919a957-a46f-4e96-b42e-3ba3c537e98e-hosts-file\") pod \"node-resolver-fjwjw\" (UID: \"2919a957-a46f-4e96-b42e-3ba3c537e98e\") " pod="openshift-dns/node-resolver-fjwjw" Oct 11 10:28:48.967219 master-1 kubenswrapper[4771]: I1011 10:28:48.967147 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") pod \"controller-manager-857df878cf-tz7h4\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:48.978207 master-1 kubenswrapper[4771]: I1011 10:28:48.978168 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpt6z\" (UniqueName: \"kubernetes.io/projected/2919a957-a46f-4e96-b42e-3ba3c537e98e-kube-api-access-zpt6z\") pod \"node-resolver-fjwjw\" (UID: \"2919a957-a46f-4e96-b42e-3ba3c537e98e\") " pod="openshift-dns/node-resolver-fjwjw" Oct 11 10:28:48.997622 master-1 kubenswrapper[4771]: I1011 10:28:48.997572 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rzjcf"] Oct 11 10:28:49.006656 master-1 kubenswrapper[4771]: W1011 10:28:49.006588 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f49f37_a9e4_4acd_ae7e_d644e8475106.slice/crio-8ef7b976a4f384167d2fc481618a09fa0aee47bd492fde332c525af91b56c920 WatchSource:0}: Error finding container 8ef7b976a4f384167d2fc481618a09fa0aee47bd492fde332c525af91b56c920: Status 404 returned error can't find the container with id 8ef7b976a4f384167d2fc481618a09fa0aee47bd492fde332c525af91b56c920 Oct 11 10:28:49.061999 master-1 kubenswrapper[4771]: I1011 10:28:49.061910 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:49.062295 master-1 kubenswrapper[4771]: E1011 10:28:49.062248 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:28:53.06218303 +0000 UTC m=+165.036409481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:49.097586 master-1 kubenswrapper[4771]: I1011 10:28:49.097463 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fjwjw" Oct 11 10:28:49.101988 master-1 kubenswrapper[4771]: I1011 10:28:49.101931 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzjcf" event={"ID":"b3f49f37-a9e4-4acd-ae7e-d644e8475106","Type":"ContainerStarted","Data":"8ef7b976a4f384167d2fc481618a09fa0aee47bd492fde332c525af91b56c920"} Oct 11 10:28:49.436717 master-1 kubenswrapper[4771]: I1011 10:28:49.436606 4771 scope.go:117] "RemoveContainer" containerID="793c72629ffb5d64763cce906980f11774530f02d707e0389b69155b33560c5d" Oct 11 10:28:49.436888 master-1 kubenswrapper[4771]: E1011 10:28:49.436797 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-rbac-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=kube-rbac-proxy pod=cluster-cloud-controller-manager-operator-779749f859-5xxzp_openshift-cloud-controller-manager-operator(e115f8be-9e65-4407-8111-568e5ea8ac1b)\"" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" podUID="e115f8be-9e65-4407-8111-568e5ea8ac1b" Oct 11 10:28:49.578448 master-1 kubenswrapper[4771]: W1011 10:28:49.578396 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2919a957_a46f_4e96_b42e_3ba3c537e98e.slice/crio-8f6998e0a5e0c5234251889b81be2d102553dd2fe4f826390029300e639e331a WatchSource:0}: Error finding container 8f6998e0a5e0c5234251889b81be2d102553dd2fe4f826390029300e639e331a: Status 404 returned error can't find the container with id 8f6998e0a5e0c5234251889b81be2d102553dd2fe4f826390029300e639e331a Oct 11 10:28:49.952480 master-1 kubenswrapper[4771]: I1011 10:28:49.951964 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-vhfgw"] Oct 11 10:28:49.953164 master-1 kubenswrapper[4771]: I1011 10:28:49.953112 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:49.955723 master-1 kubenswrapper[4771]: I1011 10:28:49.955658 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cluster-node-tuning-operator"/"openshift-service-ca.crt" Oct 11 10:28:49.956442 master-1 kubenswrapper[4771]: I1011 10:28:49.956382 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cluster-node-tuning-operator"/"kube-root-ca.crt" Oct 11 10:28:50.083654 master-1 kubenswrapper[4771]: I1011 10:28:50.083507 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-systemd\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.083654 master-1 kubenswrapper[4771]: I1011 10:28:50.083658 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-kubernetes\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084014 master-1 kubenswrapper[4771]: I1011 10:28:50.083711 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-sysctl-d\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084014 master-1 kubenswrapper[4771]: I1011 10:28:50.083826 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5a3f75a-c5b4-407a-b16a-5277aec051f7-tmp\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084014 master-1 kubenswrapper[4771]: I1011 10:28:50.083879 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-modprobe-d\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084014 master-1 kubenswrapper[4771]: I1011 10:28:50.083924 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-sysconfig\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084292 master-1 kubenswrapper[4771]: I1011 10:28:50.084100 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-lib-modules\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084292 master-1 kubenswrapper[4771]: I1011 10:28:50.084226 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-run\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084483 master-1 kubenswrapper[4771]: I1011 10:28:50.084316 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-host\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084558 master-1 kubenswrapper[4771]: I1011 10:28:50.084534 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clwnw\" (UniqueName: \"kubernetes.io/projected/f5a3f75a-c5b4-407a-b16a-5277aec051f7-kube-api-access-clwnw\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084623 master-1 kubenswrapper[4771]: I1011 10:28:50.084597 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-sysctl-conf\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084742 master-1 kubenswrapper[4771]: I1011 10:28:50.084627 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-tuned\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084810 master-1 kubenswrapper[4771]: I1011 10:28:50.084779 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-sys\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.084946 master-1 kubenswrapper[4771]: I1011 10:28:50.084896 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-var-lib-kubelet\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.107563 master-1 kubenswrapper[4771]: I1011 10:28:50.107479 4771 generic.go:334] "Generic (PLEG): container finished" podID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerID="e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346" exitCode=0 Oct 11 10:28:50.107563 master-1 kubenswrapper[4771]: I1011 10:28:50.107561 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" event={"ID":"004ee387-d0e9-4582-ad14-f571832ebd6e","Type":"ContainerDied","Data":"e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346"} Oct 11 10:28:50.110026 master-1 kubenswrapper[4771]: I1011 10:28:50.109961 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fjwjw" event={"ID":"2919a957-a46f-4e96-b42e-3ba3c537e98e","Type":"ContainerStarted","Data":"0ed94446953dd34ae6187c546beb5e48bd72be4a6a0fb6994d62b9be96b82a01"} Oct 11 10:28:50.110026 master-1 kubenswrapper[4771]: I1011 10:28:50.110026 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fjwjw" event={"ID":"2919a957-a46f-4e96-b42e-3ba3c537e98e","Type":"ContainerStarted","Data":"8f6998e0a5e0c5234251889b81be2d102553dd2fe4f826390029300e639e331a"} Oct 11 10:28:50.125933 master-1 kubenswrapper[4771]: I1011 10:28:50.125866 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-857df878cf-tz7h4"] Oct 11 10:28:50.126142 master-1 kubenswrapper[4771]: E1011 10:28:50.126095 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" podUID="9deef4a8-bf40-4a1f-bd3f-764b298245b2" Oct 11 10:28:50.142464 master-1 kubenswrapper[4771]: I1011 10:28:50.142387 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf"] Oct 11 10:28:50.142598 master-1 kubenswrapper[4771]: E1011 10:28:50.142561 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" podUID="67be9a32-17b4-480c-98ba-caf9841bef6b" Oct 11 10:28:50.169945 master-1 kubenswrapper[4771]: I1011 10:28:50.169809 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-fjwjw" podStartSLOduration=2.169783123 podStartE2EDuration="2.169783123s" podCreationTimestamp="2025-10-11 10:28:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:28:50.167481332 +0000 UTC m=+162.141707813" watchObservedRunningTime="2025-10-11 10:28:50.169783123 +0000 UTC m=+162.144009604" Oct 11 10:28:50.185858 master-1 kubenswrapper[4771]: I1011 10:28:50.185771 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-systemd\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.185858 master-1 kubenswrapper[4771]: I1011 10:28:50.185861 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-kubernetes\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186065 master-1 kubenswrapper[4771]: I1011 10:28:50.185881 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-sysctl-d\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186065 master-1 kubenswrapper[4771]: I1011 10:28:50.185909 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5a3f75a-c5b4-407a-b16a-5277aec051f7-tmp\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186065 master-1 kubenswrapper[4771]: I1011 10:28:50.185925 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-modprobe-d\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186065 master-1 kubenswrapper[4771]: I1011 10:28:50.185918 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-systemd\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186065 master-1 kubenswrapper[4771]: I1011 10:28:50.185939 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-sysconfig\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186065 master-1 kubenswrapper[4771]: I1011 10:28:50.186004 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-sysconfig\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186065 master-1 kubenswrapper[4771]: I1011 10:28:50.186027 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-lib-modules\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186065 master-1 kubenswrapper[4771]: I1011 10:28:50.186069 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-run\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186559 master-1 kubenswrapper[4771]: I1011 10:28:50.186089 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-sysctl-d\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186559 master-1 kubenswrapper[4771]: I1011 10:28:50.186149 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-host\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186559 master-1 kubenswrapper[4771]: I1011 10:28:50.186241 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-kubernetes\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186559 master-1 kubenswrapper[4771]: I1011 10:28:50.186327 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-lib-modules\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186559 master-1 kubenswrapper[4771]: I1011 10:28:50.186377 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-run\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186559 master-1 kubenswrapper[4771]: I1011 10:28:50.186467 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-host\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186930 master-1 kubenswrapper[4771]: I1011 10:28:50.186562 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-modprobe-d\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186930 master-1 kubenswrapper[4771]: I1011 10:28:50.186758 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-clwnw\" (UniqueName: \"kubernetes.io/projected/f5a3f75a-c5b4-407a-b16a-5277aec051f7-kube-api-access-clwnw\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186930 master-1 kubenswrapper[4771]: I1011 10:28:50.186810 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-tuned\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.186930 master-1 kubenswrapper[4771]: I1011 10:28:50.186873 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-sysctl-conf\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.187231 master-1 kubenswrapper[4771]: I1011 10:28:50.186969 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-sys\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.187231 master-1 kubenswrapper[4771]: I1011 10:28:50.187027 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-var-lib-kubelet\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.187231 master-1 kubenswrapper[4771]: I1011 10:28:50.187128 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-var-lib-kubelet\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.187231 master-1 kubenswrapper[4771]: I1011 10:28:50.187203 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-sys\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.187550 master-1 kubenswrapper[4771]: I1011 10:28:50.187278 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-sysctl-conf\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.190806 master-1 kubenswrapper[4771]: I1011 10:28:50.190749 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f5a3f75a-c5b4-407a-b16a-5277aec051f7-etc-tuned\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.190955 master-1 kubenswrapper[4771]: I1011 10:28:50.190758 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5a3f75a-c5b4-407a-b16a-5277aec051f7-tmp\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.207388 master-1 kubenswrapper[4771]: I1011 10:28:50.207294 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-clwnw\" (UniqueName: \"kubernetes.io/projected/f5a3f75a-c5b4-407a-b16a-5277aec051f7-kube-api-access-clwnw\") pod \"tuned-vhfgw\" (UID: \"f5a3f75a-c5b4-407a-b16a-5277aec051f7\") " pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.267507 master-1 kubenswrapper[4771]: I1011 10:28:50.267442 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" Oct 11 10:28:50.277785 master-1 kubenswrapper[4771]: W1011 10:28:50.277656 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a3f75a_c5b4_407a_b16a_5277aec051f7.slice/crio-f54666226746504bded08b7b482b73574d5dbca7369c120b64bc771746344af4 WatchSource:0}: Error finding container f54666226746504bded08b7b482b73574d5dbca7369c120b64bc771746344af4: Status 404 returned error can't find the container with id f54666226746504bded08b7b482b73574d5dbca7369c120b64bc771746344af4 Oct 11 10:28:51.093325 master-1 kubenswrapper[4771]: I1011 10:28:51.092919 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-scheduler/installer-1-master-1"] Oct 11 10:28:51.093521 master-1 kubenswrapper[4771]: I1011 10:28:51.093344 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-scheduler/installer-1-master-1" podUID="007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7" containerName="installer" containerID="cri-o://80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90" gracePeriod=30 Oct 11 10:28:51.116010 master-1 kubenswrapper[4771]: I1011 10:28:51.115910 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" event={"ID":"f5a3f75a-c5b4-407a-b16a-5277aec051f7","Type":"ContainerStarted","Data":"f54666226746504bded08b7b482b73574d5dbca7369c120b64bc771746344af4"} Oct 11 10:28:51.118699 master-1 kubenswrapper[4771]: I1011 10:28:51.118641 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:51.119585 master-1 kubenswrapper[4771]: I1011 10:28:51.119510 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:51.119585 master-1 kubenswrapper[4771]: I1011 10:28:51.119562 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" event={"ID":"004ee387-d0e9-4582-ad14-f571832ebd6e","Type":"ContainerStarted","Data":"913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68"} Oct 11 10:28:51.128373 master-1 kubenswrapper[4771]: I1011 10:28:51.128294 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:51.133766 master-1 kubenswrapper[4771]: I1011 10:28:51.133721 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:51.139382 master-1 kubenswrapper[4771]: I1011 10:28:51.139288 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podStartSLOduration=6.100808295 podStartE2EDuration="8.13926696s" podCreationTimestamp="2025-10-11 10:28:43 +0000 UTC" firstStartedPulling="2025-10-11 10:28:47.569445427 +0000 UTC m=+159.543671858" lastFinishedPulling="2025-10-11 10:28:49.607904042 +0000 UTC m=+161.582130523" observedRunningTime="2025-10-11 10:28:51.138796377 +0000 UTC m=+163.113022848" watchObservedRunningTime="2025-10-11 10:28:51.13926696 +0000 UTC m=+163.113493441" Oct 11 10:28:51.199556 master-1 kubenswrapper[4771]: I1011 10:28:51.199448 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-config\") pod \"67be9a32-17b4-480c-98ba-caf9841bef6b\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " Oct 11 10:28:51.199733 master-1 kubenswrapper[4771]: I1011 10:28:51.199603 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") pod \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " Oct 11 10:28:51.200337 master-1 kubenswrapper[4771]: I1011 10:28:51.200280 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-config" (OuterVolumeSpecName: "config") pod "67be9a32-17b4-480c-98ba-caf9841bef6b" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:28:51.200945 master-1 kubenswrapper[4771]: I1011 10:28:51.199729 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") pod \"67be9a32-17b4-480c-98ba-caf9841bef6b\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " Oct 11 10:28:51.201103 master-1 kubenswrapper[4771]: I1011 10:28:51.201033 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-proxy-ca-bundles\") pod \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " Oct 11 10:28:51.202375 master-1 kubenswrapper[4771]: I1011 10:28:51.201928 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-proxy-ca-bundles" (OuterVolumeSpecName: "proxy-ca-bundles") pod "9deef4a8-bf40-4a1f-bd3f-764b298245b2" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2"). InnerVolumeSpecName "proxy-ca-bundles". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:28:51.203324 master-1 kubenswrapper[4771]: I1011 10:28:51.203276 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hktq9\" (UniqueName: \"kubernetes.io/projected/9deef4a8-bf40-4a1f-bd3f-764b298245b2-kube-api-access-hktq9\") pod \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " Oct 11 10:28:51.203455 master-1 kubenswrapper[4771]: I1011 10:28:51.203421 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-config\") pod \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\" (UID: \"9deef4a8-bf40-4a1f-bd3f-764b298245b2\") " Oct 11 10:28:51.203543 master-1 kubenswrapper[4771]: I1011 10:28:51.203512 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcj47\" (UniqueName: \"kubernetes.io/projected/67be9a32-17b4-480c-98ba-caf9841bef6b-kube-api-access-fcj47\") pod \"67be9a32-17b4-480c-98ba-caf9841bef6b\" (UID: \"67be9a32-17b4-480c-98ba-caf9841bef6b\") " Oct 11 10:28:51.204313 master-1 kubenswrapper[4771]: I1011 10:28:51.204254 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-config" (OuterVolumeSpecName: "config") pod "9deef4a8-bf40-4a1f-bd3f-764b298245b2" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:28:51.204438 master-1 kubenswrapper[4771]: I1011 10:28:51.204391 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "9deef4a8-bf40-4a1f-bd3f-764b298245b2" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:28:51.204948 master-1 kubenswrapper[4771]: I1011 10:28:51.204892 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:51.205242 master-1 kubenswrapper[4771]: I1011 10:28:51.205213 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9deef4a8-bf40-4a1f-bd3f-764b298245b2-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:51.205407 master-1 kubenswrapper[4771]: I1011 10:28:51.205384 4771 reconciler_common.go:293] "Volume detached for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-proxy-ca-bundles\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:51.205566 master-1 kubenswrapper[4771]: I1011 10:28:51.205545 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:51.205722 master-1 kubenswrapper[4771]: I1011 10:28:51.205214 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "67be9a32-17b4-480c-98ba-caf9841bef6b" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:28:51.206403 master-1 kubenswrapper[4771]: I1011 10:28:51.206320 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67be9a32-17b4-480c-98ba-caf9841bef6b-kube-api-access-fcj47" (OuterVolumeSpecName: "kube-api-access-fcj47") pod "67be9a32-17b4-480c-98ba-caf9841bef6b" (UID: "67be9a32-17b4-480c-98ba-caf9841bef6b"). InnerVolumeSpecName "kube-api-access-fcj47". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:28:51.206954 master-1 kubenswrapper[4771]: I1011 10:28:51.206889 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deef4a8-bf40-4a1f-bd3f-764b298245b2-kube-api-access-hktq9" (OuterVolumeSpecName: "kube-api-access-hktq9") pod "9deef4a8-bf40-4a1f-bd3f-764b298245b2" (UID: "9deef4a8-bf40-4a1f-bd3f-764b298245b2"). InnerVolumeSpecName "kube-api-access-hktq9". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:28:51.306858 master-1 kubenswrapper[4771]: I1011 10:28:51.306792 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67be9a32-17b4-480c-98ba-caf9841bef6b-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:51.306858 master-1 kubenswrapper[4771]: I1011 10:28:51.306848 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-hktq9\" (UniqueName: \"kubernetes.io/projected/9deef4a8-bf40-4a1f-bd3f-764b298245b2-kube-api-access-hktq9\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:51.307061 master-1 kubenswrapper[4771]: I1011 10:28:51.306869 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-fcj47\" (UniqueName: \"kubernetes.io/projected/67be9a32-17b4-480c-98ba-caf9841bef6b-kube-api-access-fcj47\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:52.065560 master-1 kubenswrapper[4771]: I1011 10:28:52.065103 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-etcd/installer-1-master-1"] Oct 11 10:28:52.066508 master-1 kubenswrapper[4771]: I1011 10:28:52.066473 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.069745 master-1 kubenswrapper[4771]: I1011 10:28:52.069693 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-etcd"/"kube-root-ca.crt" Oct 11 10:28:52.073024 master-1 kubenswrapper[4771]: I1011 10:28:52.072964 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/installer-1-master-1"] Oct 11 10:28:52.125614 master-1 kubenswrapper[4771]: I1011 10:28:52.125520 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzjcf" event={"ID":"b3f49f37-a9e4-4acd-ae7e-d644e8475106","Type":"ContainerStarted","Data":"acafd6bd0153153d6b38fbdc6317872f00d21dba7abb8bb24592c18f1e0a2729"} Oct 11 10:28:52.125614 master-1 kubenswrapper[4771]: I1011 10:28:52.125607 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzjcf" event={"ID":"b3f49f37-a9e4-4acd-ae7e-d644e8475106","Type":"ContainerStarted","Data":"4baab67efc4d26139295ca37e7c077eab06b8a73e0c12560e14c2c4b7a1656ca"} Oct 11 10:28:52.125614 master-1 kubenswrapper[4771]: I1011 10:28:52.125566 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-857df878cf-tz7h4" Oct 11 10:28:52.126420 master-1 kubenswrapper[4771]: I1011 10:28:52.125718 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf" Oct 11 10:28:52.152609 master-1 kubenswrapper[4771]: I1011 10:28:52.152541 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-rzjcf" podStartSLOduration=2.096689283 podStartE2EDuration="4.152511418s" podCreationTimestamp="2025-10-11 10:28:48 +0000 UTC" firstStartedPulling="2025-10-11 10:28:49.01319373 +0000 UTC m=+160.987420181" lastFinishedPulling="2025-10-11 10:28:51.069015835 +0000 UTC m=+163.043242316" observedRunningTime="2025-10-11 10:28:52.152310253 +0000 UTC m=+164.126536764" watchObservedRunningTime="2025-10-11 10:28:52.152511418 +0000 UTC m=+164.126737859" Oct 11 10:28:52.176386 master-1 kubenswrapper[4771]: I1011 10:28:52.176305 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw"] Oct 11 10:28:52.177752 master-1 kubenswrapper[4771]: I1011 10:28:52.177684 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.180377 master-1 kubenswrapper[4771]: I1011 10:28:52.180306 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"config" Oct 11 10:28:52.181041 master-1 kubenswrapper[4771]: I1011 10:28:52.180984 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"client-ca" Oct 11 10:28:52.181605 master-1 kubenswrapper[4771]: I1011 10:28:52.181498 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf"] Oct 11 10:28:52.181605 master-1 kubenswrapper[4771]: I1011 10:28:52.181507 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"openshift-service-ca.crt" Oct 11 10:28:52.181820 master-1 kubenswrapper[4771]: I1011 10:28:52.181300 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"kube-root-ca.crt" Oct 11 10:28:52.181820 master-1 kubenswrapper[4771]: I1011 10:28:52.181440 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-route-controller-manager"/"serving-cert" Oct 11 10:28:52.183755 master-1 kubenswrapper[4771]: I1011 10:28:52.183677 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-67d4d4d6d8-szbpf"] Oct 11 10:28:52.185469 master-1 kubenswrapper[4771]: I1011 10:28:52.185426 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw"] Oct 11 10:28:52.201874 master-1 kubenswrapper[4771]: I1011 10:28:52.201823 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-857df878cf-tz7h4"] Oct 11 10:28:52.204132 master-1 kubenswrapper[4771]: I1011 10:28:52.204062 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-controller-manager/controller-manager-857df878cf-tz7h4"] Oct 11 10:28:52.217654 master-1 kubenswrapper[4771]: I1011 10:28:52.217603 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/826e1279-bc0d-426e-b6e0-5108268f340e-kube-api-access\") pod \"installer-1-master-1\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.217654 master-1 kubenswrapper[4771]: I1011 10:28:52.217660 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.217969 master-1 kubenswrapper[4771]: I1011 10:28:52.217679 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-var-lock\") pod \"installer-1-master-1\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.318870 master-1 kubenswrapper[4771]: I1011 10:28:52.318699 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwxw7\" (UniqueName: \"kubernetes.io/projected/537a2b50-0394-47bd-941a-def350316943-kube-api-access-zwxw7\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.318870 master-1 kubenswrapper[4771]: I1011 10:28:52.318747 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.318870 master-1 kubenswrapper[4771]: I1011 10:28:52.318794 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/826e1279-bc0d-426e-b6e0-5108268f340e-kube-api-access\") pod \"installer-1-master-1\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.318870 master-1 kubenswrapper[4771]: I1011 10:28:52.318820 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.318870 master-1 kubenswrapper[4771]: I1011 10:28:52.318842 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/537a2b50-0394-47bd-941a-def350316943-serving-cert\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.318870 master-1 kubenswrapper[4771]: I1011 10:28:52.318864 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-var-lock\") pod \"installer-1-master-1\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.319246 master-1 kubenswrapper[4771]: I1011 10:28:52.318924 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-config\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.319246 master-1 kubenswrapper[4771]: I1011 10:28:52.318964 4771 reconciler_common.go:293] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/67be9a32-17b4-480c-98ba-caf9841bef6b-client-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:52.319246 master-1 kubenswrapper[4771]: I1011 10:28:52.318977 4771 reconciler_common.go:293] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9deef4a8-bf40-4a1f-bd3f-764b298245b2-client-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:52.319440 master-1 kubenswrapper[4771]: I1011 10:28:52.319416 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-var-lock\") pod \"installer-1-master-1\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.319577 master-1 kubenswrapper[4771]: I1011 10:28:52.319463 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.340742 master-1 kubenswrapper[4771]: I1011 10:28:52.340684 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:52.340815 master-1 kubenswrapper[4771]: I1011 10:28:52.340760 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:52.350547 master-1 kubenswrapper[4771]: I1011 10:28:52.350430 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/826e1279-bc0d-426e-b6e0-5108268f340e-kube-api-access\") pod \"installer-1-master-1\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.353003 master-1 kubenswrapper[4771]: I1011 10:28:52.352927 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:52.381790 master-1 kubenswrapper[4771]: I1011 10:28:52.381714 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-1-master-1" Oct 11 10:28:52.446395 master-1 kubenswrapper[4771]: I1011 10:28:52.432572 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-config\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.446395 master-1 kubenswrapper[4771]: I1011 10:28:52.432711 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-zwxw7\" (UniqueName: \"kubernetes.io/projected/537a2b50-0394-47bd-941a-def350316943-kube-api-access-zwxw7\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.446395 master-1 kubenswrapper[4771]: I1011 10:28:52.432803 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.446395 master-1 kubenswrapper[4771]: I1011 10:28:52.432893 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/537a2b50-0394-47bd-941a-def350316943-serving-cert\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.446395 master-1 kubenswrapper[4771]: E1011 10:28:52.435416 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:52.446395 master-1 kubenswrapper[4771]: I1011 10:28:52.435501 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-config\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.446395 master-1 kubenswrapper[4771]: E1011 10:28:52.435575 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:52.935505058 +0000 UTC m=+164.909731529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:28:52.454386 master-1 kubenswrapper[4771]: I1011 10:28:52.453003 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/537a2b50-0394-47bd-941a-def350316943-serving-cert\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.458796 master-1 kubenswrapper[4771]: I1011 10:28:52.458723 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67be9a32-17b4-480c-98ba-caf9841bef6b" path="/var/lib/kubelet/pods/67be9a32-17b4-480c-98ba-caf9841bef6b/volumes" Oct 11 10:28:52.459443 master-1 kubenswrapper[4771]: I1011 10:28:52.459399 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9deef4a8-bf40-4a1f-bd3f-764b298245b2" path="/var/lib/kubelet/pods/9deef4a8-bf40-4a1f-bd3f-764b298245b2/volumes" Oct 11 10:28:52.469640 master-1 kubenswrapper[4771]: I1011 10:28:52.469565 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwxw7\" (UniqueName: \"kubernetes.io/projected/537a2b50-0394-47bd-941a-def350316943-kube-api-access-zwxw7\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.659836 master-1 kubenswrapper[4771]: I1011 10:28:52.659772 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/installer-1-master-1"] Oct 11 10:28:52.670195 master-1 kubenswrapper[4771]: W1011 10:28:52.670144 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod826e1279_bc0d_426e_b6e0_5108268f340e.slice/crio-5363fbd5b12a9230e5f6b1dd57b8fb070fa12eb536a76d7bdfd11f7b2167cad9 WatchSource:0}: Error finding container 5363fbd5b12a9230e5f6b1dd57b8fb070fa12eb536a76d7bdfd11f7b2167cad9: Status 404 returned error can't find the container with id 5363fbd5b12a9230e5f6b1dd57b8fb070fa12eb536a76d7bdfd11f7b2167cad9 Oct 11 10:28:52.938927 master-1 kubenswrapper[4771]: I1011 10:28:52.938668 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:52.938927 master-1 kubenswrapper[4771]: E1011 10:28:52.938806 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:52.938927 master-1 kubenswrapper[4771]: E1011 10:28:52.938876 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:53.938859926 +0000 UTC m=+165.913086377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:28:53.039956 master-1 kubenswrapper[4771]: I1011 10:28:53.039861 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:28:53.039956 master-1 kubenswrapper[4771]: E1011 10:28:53.039988 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:29:01.039972209 +0000 UTC m=+173.014198650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:53.131957 master-1 kubenswrapper[4771]: I1011 10:28:53.131876 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-1-master-1" event={"ID":"826e1279-bc0d-426e-b6e0-5108268f340e","Type":"ContainerStarted","Data":"5363fbd5b12a9230e5f6b1dd57b8fb070fa12eb536a76d7bdfd11f7b2167cad9"} Oct 11 10:28:53.134207 master-1 kubenswrapper[4771]: I1011 10:28:53.132472 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-dns/dns-default-rzjcf" Oct 11 10:28:53.137959 master-1 kubenswrapper[4771]: I1011 10:28:53.137775 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:28:53.140834 master-1 kubenswrapper[4771]: I1011 10:28:53.140785 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:28:53.141070 master-1 kubenswrapper[4771]: E1011 10:28:53.141032 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:29:01.1410091 +0000 UTC m=+173.115235561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:28:53.688506 master-1 kubenswrapper[4771]: I1011 10:28:53.688449 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/installer-2-master-1"] Oct 11 10:28:53.688865 master-1 kubenswrapper[4771]: I1011 10:28:53.688838 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:53.695212 master-1 kubenswrapper[4771]: I1011 10:28:53.694900 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-2-master-1"] Oct 11 10:28:53.847208 master-1 kubenswrapper[4771]: I1011 10:28:53.847140 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-var-lock\") pod \"installer-2-master-1\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:53.847535 master-1 kubenswrapper[4771]: I1011 10:28:53.847236 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-kubelet-dir\") pod \"installer-2-master-1\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:53.847535 master-1 kubenswrapper[4771]: I1011 10:28:53.847295 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/750e7efe-07f7-4280-85c7-78250178f965-kube-api-access\") pod \"installer-2-master-1\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:53.949014 master-1 kubenswrapper[4771]: I1011 10:28:53.948837 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-var-lock\") pod \"installer-2-master-1\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:53.949014 master-1 kubenswrapper[4771]: I1011 10:28:53.948933 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-kubelet-dir\") pod \"installer-2-master-1\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:53.949014 master-1 kubenswrapper[4771]: I1011 10:28:53.948976 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/750e7efe-07f7-4280-85c7-78250178f965-kube-api-access\") pod \"installer-2-master-1\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:53.949014 master-1 kubenswrapper[4771]: I1011 10:28:53.948997 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:53.949014 master-1 kubenswrapper[4771]: I1011 10:28:53.949005 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-var-lock\") pod \"installer-2-master-1\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:53.949429 master-1 kubenswrapper[4771]: E1011 10:28:53.949098 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:53.949429 master-1 kubenswrapper[4771]: I1011 10:28:53.949123 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-kubelet-dir\") pod \"installer-2-master-1\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:53.949429 master-1 kubenswrapper[4771]: E1011 10:28:53.949153 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:55.949139496 +0000 UTC m=+167.923365937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:28:53.974804 master-1 kubenswrapper[4771]: I1011 10:28:53.974713 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/750e7efe-07f7-4280-85c7-78250178f965-kube-api-access\") pod \"installer-2-master-1\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:54.005417 master-1 kubenswrapper[4771]: I1011 10:28:54.005344 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:28:54.614671 master-1 kubenswrapper[4771]: I1011 10:28:54.614601 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-controller-manager/controller-manager-565f857764-nhm4g"] Oct 11 10:28:54.615343 master-1 kubenswrapper[4771]: I1011 10:28:54.615308 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.618089 master-1 kubenswrapper[4771]: I1011 10:28:54.618049 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"config" Oct 11 10:28:54.619428 master-1 kubenswrapper[4771]: I1011 10:28:54.619391 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"client-ca" Oct 11 10:28:54.619478 master-1 kubenswrapper[4771]: I1011 10:28:54.619464 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"kube-root-ca.crt" Oct 11 10:28:54.619826 master-1 kubenswrapper[4771]: I1011 10:28:54.619744 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-service-ca.crt" Oct 11 10:28:54.620839 master-1 kubenswrapper[4771]: I1011 10:28:54.620808 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-controller-manager"/"serving-cert" Oct 11 10:28:54.628523 master-1 kubenswrapper[4771]: I1011 10:28:54.628461 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-controller-manager/controller-manager-565f857764-nhm4g"] Oct 11 10:28:54.631782 master-1 kubenswrapper[4771]: I1011 10:28:54.631713 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-global-ca" Oct 11 10:28:54.757422 master-1 kubenswrapper[4771]: I1011 10:28:54.757334 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.757422 master-1 kubenswrapper[4771]: I1011 10:28:54.757426 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-proxy-ca-bundles\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.757669 master-1 kubenswrapper[4771]: I1011 10:28:54.757482 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9e9455e-0b47-4623-9b4c-ef79cf62a254-serving-cert\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.757669 master-1 kubenswrapper[4771]: I1011 10:28:54.757520 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wp47\" (UniqueName: \"kubernetes.io/projected/c9e9455e-0b47-4623-9b4c-ef79cf62a254-kube-api-access-9wp47\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.757669 master-1 kubenswrapper[4771]: I1011 10:28:54.757558 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-config\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.858189 master-1 kubenswrapper[4771]: I1011 10:28:54.858115 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.858189 master-1 kubenswrapper[4771]: I1011 10:28:54.858171 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-proxy-ca-bundles\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.858189 master-1 kubenswrapper[4771]: I1011 10:28:54.858198 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9e9455e-0b47-4623-9b4c-ef79cf62a254-serving-cert\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.858541 master-1 kubenswrapper[4771]: I1011 10:28:54.858221 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-9wp47\" (UniqueName: \"kubernetes.io/projected/c9e9455e-0b47-4623-9b4c-ef79cf62a254-kube-api-access-9wp47\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.858541 master-1 kubenswrapper[4771]: I1011 10:28:54.858245 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-config\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.860666 master-1 kubenswrapper[4771]: E1011 10:28:54.859600 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:54.860666 master-1 kubenswrapper[4771]: E1011 10:28:54.859761 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:55.359705869 +0000 UTC m=+167.333932350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:28:54.863432 master-1 kubenswrapper[4771]: I1011 10:28:54.862122 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-config\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.866529 master-1 kubenswrapper[4771]: I1011 10:28:54.864615 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9e9455e-0b47-4623-9b4c-ef79cf62a254-serving-cert\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.866529 master-1 kubenswrapper[4771]: I1011 10:28:54.865914 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-proxy-ca-bundles\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:54.877715 master-1 kubenswrapper[4771]: I1011 10:28:54.877639 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wp47\" (UniqueName: \"kubernetes.io/projected/c9e9455e-0b47-4623-9b4c-ef79cf62a254-kube-api-access-9wp47\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:55.366554 master-1 kubenswrapper[4771]: I1011 10:28:55.366459 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:55.366755 master-1 kubenswrapper[4771]: E1011 10:28:55.366715 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:55.366883 master-1 kubenswrapper[4771]: E1011 10:28:55.366853 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:56.366825207 +0000 UTC m=+168.341051688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:28:55.973802 master-1 kubenswrapper[4771]: I1011 10:28:55.973702 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:28:55.974613 master-1 kubenswrapper[4771]: E1011 10:28:55.973916 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:55.974613 master-1 kubenswrapper[4771]: E1011 10:28:55.974012 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:59.97398837 +0000 UTC m=+171.948214841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:28:56.347663 master-1 kubenswrapper[4771]: I1011 10:28:56.347617 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-2-master-1"] Oct 11 10:28:56.362316 master-1 kubenswrapper[4771]: W1011 10:28:56.362270 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod750e7efe_07f7_4280_85c7_78250178f965.slice/crio-23a2788a32f3ff82c23ee63b5bf39839725cd21b8f03a9a69d479fa53764ca28 WatchSource:0}: Error finding container 23a2788a32f3ff82c23ee63b5bf39839725cd21b8f03a9a69d479fa53764ca28: Status 404 returned error can't find the container with id 23a2788a32f3ff82c23ee63b5bf39839725cd21b8f03a9a69d479fa53764ca28 Oct 11 10:28:56.378810 master-1 kubenswrapper[4771]: I1011 10:28:56.378749 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:56.378949 master-1 kubenswrapper[4771]: E1011 10:28:56.378882 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:56.379015 master-1 kubenswrapper[4771]: E1011 10:28:56.378986 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:28:58.378957656 +0000 UTC m=+170.353184117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:28:56.656433 master-1 kubenswrapper[4771]: I1011 10:28:56.655876 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4"] Oct 11 10:28:56.657788 master-1 kubenswrapper[4771]: I1011 10:28:56.657749 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" Oct 11 10:28:56.661131 master-1 kubenswrapper[4771]: I1011 10:28:56.661053 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cluster-samples-operator"/"kube-root-ca.crt" Oct 11 10:28:56.662005 master-1 kubenswrapper[4771]: I1011 10:28:56.661941 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-cluster-samples-operator"/"samples-operator-tls" Oct 11 10:28:56.662137 master-1 kubenswrapper[4771]: I1011 10:28:56.662006 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-cluster-samples-operator"/"openshift-service-ca.crt" Oct 11 10:28:56.667238 master-1 kubenswrapper[4771]: I1011 10:28:56.667165 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4"] Oct 11 10:28:56.783045 master-1 kubenswrapper[4771]: I1011 10:28:56.782860 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b763cbe4-f035-45f2-9f70-4bbb8d5cac87-samples-operator-tls\") pod \"cluster-samples-operator-75f9c7d795-2zgv4\" (UID: \"b763cbe4-f035-45f2-9f70-4bbb8d5cac87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" Oct 11 10:28:56.783045 master-1 kubenswrapper[4771]: I1011 10:28:56.783006 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffrmg\" (UniqueName: \"kubernetes.io/projected/b763cbe4-f035-45f2-9f70-4bbb8d5cac87-kube-api-access-ffrmg\") pod \"cluster-samples-operator-75f9c7d795-2zgv4\" (UID: \"b763cbe4-f035-45f2-9f70-4bbb8d5cac87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" Oct 11 10:28:56.884718 master-1 kubenswrapper[4771]: I1011 10:28:56.884642 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b763cbe4-f035-45f2-9f70-4bbb8d5cac87-samples-operator-tls\") pod \"cluster-samples-operator-75f9c7d795-2zgv4\" (UID: \"b763cbe4-f035-45f2-9f70-4bbb8d5cac87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" Oct 11 10:28:56.884973 master-1 kubenswrapper[4771]: I1011 10:28:56.884806 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-ffrmg\" (UniqueName: \"kubernetes.io/projected/b763cbe4-f035-45f2-9f70-4bbb8d5cac87-kube-api-access-ffrmg\") pod \"cluster-samples-operator-75f9c7d795-2zgv4\" (UID: \"b763cbe4-f035-45f2-9f70-4bbb8d5cac87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" Oct 11 10:28:56.892511 master-1 kubenswrapper[4771]: I1011 10:28:56.892436 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b763cbe4-f035-45f2-9f70-4bbb8d5cac87-samples-operator-tls\") pod \"cluster-samples-operator-75f9c7d795-2zgv4\" (UID: \"b763cbe4-f035-45f2-9f70-4bbb8d5cac87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" Oct 11 10:28:56.908100 master-1 kubenswrapper[4771]: I1011 10:28:56.908015 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffrmg\" (UniqueName: \"kubernetes.io/projected/b763cbe4-f035-45f2-9f70-4bbb8d5cac87-kube-api-access-ffrmg\") pod \"cluster-samples-operator-75f9c7d795-2zgv4\" (UID: \"b763cbe4-f035-45f2-9f70-4bbb8d5cac87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" Oct 11 10:28:56.979240 master-1 kubenswrapper[4771]: I1011 10:28:56.979157 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" Oct 11 10:28:57.150850 master-1 kubenswrapper[4771]: I1011 10:28:57.150768 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" event={"ID":"f5a3f75a-c5b4-407a-b16a-5277aec051f7","Type":"ContainerStarted","Data":"f9a8c29368a9534369890a79857333416edb6b50b525db38508428bb0d6a4590"} Oct 11 10:28:57.153883 master-1 kubenswrapper[4771]: I1011 10:28:57.153816 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-1-master-1" event={"ID":"826e1279-bc0d-426e-b6e0-5108268f340e","Type":"ContainerStarted","Data":"9a616ae6ac6ffcbc27ae54a54aec1c65046926d3773ee73ab8bfdedb75371f06"} Oct 11 10:28:57.155931 master-1 kubenswrapper[4771]: I1011 10:28:57.155869 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-2-master-1" event={"ID":"750e7efe-07f7-4280-85c7-78250178f965","Type":"ContainerStarted","Data":"2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2"} Oct 11 10:28:57.156018 master-1 kubenswrapper[4771]: I1011 10:28:57.155934 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-2-master-1" event={"ID":"750e7efe-07f7-4280-85c7-78250178f965","Type":"ContainerStarted","Data":"23a2788a32f3ff82c23ee63b5bf39839725cd21b8f03a9a69d479fa53764ca28"} Oct 11 10:28:57.170791 master-1 kubenswrapper[4771]: I1011 10:28:57.169653 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vhfgw" podStartSLOduration=2.324017334 podStartE2EDuration="8.169629989s" podCreationTimestamp="2025-10-11 10:28:49 +0000 UTC" firstStartedPulling="2025-10-11 10:28:50.27930851 +0000 UTC m=+162.253534991" lastFinishedPulling="2025-10-11 10:28:56.124921205 +0000 UTC m=+168.099147646" observedRunningTime="2025-10-11 10:28:57.16891912 +0000 UTC m=+169.143145591" watchObservedRunningTime="2025-10-11 10:28:57.169629989 +0000 UTC m=+169.143856470" Oct 11 10:28:57.184117 master-1 kubenswrapper[4771]: I1011 10:28:57.184042 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/installer-2-master-1" podStartSLOduration=4.184021791 podStartE2EDuration="4.184021791s" podCreationTimestamp="2025-10-11 10:28:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:28:57.182475979 +0000 UTC m=+169.156702480" watchObservedRunningTime="2025-10-11 10:28:57.184021791 +0000 UTC m=+169.158248272" Oct 11 10:28:57.215931 master-1 kubenswrapper[4771]: I1011 10:28:57.215793 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-etcd/installer-1-master-1" podStartSLOduration=1.740002676 podStartE2EDuration="5.215765223s" podCreationTimestamp="2025-10-11 10:28:52 +0000 UTC" firstStartedPulling="2025-10-11 10:28:52.67410995 +0000 UTC m=+164.648336391" lastFinishedPulling="2025-10-11 10:28:56.149872487 +0000 UTC m=+168.124098938" observedRunningTime="2025-10-11 10:28:57.199578853 +0000 UTC m=+169.173805314" watchObservedRunningTime="2025-10-11 10:28:57.215765223 +0000 UTC m=+169.189991694" Oct 11 10:28:57.216662 master-1 kubenswrapper[4771]: I1011 10:28:57.216241 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4"] Oct 11 10:28:58.029813 master-1 kubenswrapper[4771]: I1011 10:28:58.029750 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-apiserver/apiserver-796c687c6d-9b677"] Oct 11 10:28:58.030549 master-1 kubenswrapper[4771]: E1011 10:28:58.030020 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[audit], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-apiserver/apiserver-796c687c6d-9b677" podUID="94d811f4-4ac9-46b0-b937-d3370b1b4305" Oct 11 10:28:58.164222 master-1 kubenswrapper[4771]: I1011 10:28:58.163767 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" event={"ID":"b763cbe4-f035-45f2-9f70-4bbb8d5cac87","Type":"ContainerStarted","Data":"19901557922a89a89ad56c45f63b913979e9a7ab1f0d9d02378b0fbc33650102"} Oct 11 10:28:58.164469 master-1 kubenswrapper[4771]: I1011 10:28:58.164238 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:58.171696 master-1 kubenswrapper[4771]: I1011 10:28:58.171662 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:58.301592 master-1 kubenswrapper[4771]: I1011 10:28:58.301531 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-serving-cert\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.301857 master-1 kubenswrapper[4771]: I1011 10:28:58.301604 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-node-pullsecrets\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.301857 master-1 kubenswrapper[4771]: I1011 10:28:58.301653 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbkqb\" (UniqueName: \"kubernetes.io/projected/94d811f4-4ac9-46b0-b937-d3370b1b4305-kube-api-access-dbkqb\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.301857 master-1 kubenswrapper[4771]: I1011 10:28:58.301716 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-serving-ca\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.301857 master-1 kubenswrapper[4771]: I1011 10:28:58.301756 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-client\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.301857 master-1 kubenswrapper[4771]: I1011 10:28:58.301785 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-image-import-ca\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.301857 master-1 kubenswrapper[4771]: I1011 10:28:58.301821 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-config\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.301857 master-1 kubenswrapper[4771]: I1011 10:28:58.301853 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-trusted-ca-bundle\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.302265 master-1 kubenswrapper[4771]: I1011 10:28:58.301887 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit-dir\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.302265 master-1 kubenswrapper[4771]: I1011 10:28:58.301929 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-encryption-config\") pod \"94d811f4-4ac9-46b0-b937-d3370b1b4305\" (UID: \"94d811f4-4ac9-46b0-b937-d3370b1b4305\") " Oct 11 10:28:58.304509 master-1 kubenswrapper[4771]: I1011 10:28:58.304324 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-node-pullsecrets" (OuterVolumeSpecName: "node-pullsecrets") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "node-pullsecrets". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:28:58.304992 master-1 kubenswrapper[4771]: I1011 10:28:58.304935 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-serving-ca" (OuterVolumeSpecName: "etcd-serving-ca") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "etcd-serving-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:28:58.305090 master-1 kubenswrapper[4771]: I1011 10:28:58.305007 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-config" (OuterVolumeSpecName: "config") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:28:58.305090 master-1 kubenswrapper[4771]: I1011 10:28:58.305042 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-image-import-ca" (OuterVolumeSpecName: "image-import-ca") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "image-import-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:28:58.305952 master-1 kubenswrapper[4771]: I1011 10:28:58.305686 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:28:58.306102 master-1 kubenswrapper[4771]: I1011 10:28:58.306050 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:28:58.308606 master-1 kubenswrapper[4771]: I1011 10:28:58.308512 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:28:58.309237 master-1 kubenswrapper[4771]: I1011 10:28:58.309195 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d811f4-4ac9-46b0-b937-d3370b1b4305-kube-api-access-dbkqb" (OuterVolumeSpecName: "kube-api-access-dbkqb") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "kube-api-access-dbkqb". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:28:58.309433 master-1 kubenswrapper[4771]: I1011 10:28:58.309345 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-client" (OuterVolumeSpecName: "etcd-client") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "etcd-client". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:28:58.310348 master-1 kubenswrapper[4771]: I1011 10:28:58.310279 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-encryption-config" (OuterVolumeSpecName: "encryption-config") pod "94d811f4-4ac9-46b0-b937-d3370b1b4305" (UID: "94d811f4-4ac9-46b0-b937-d3370b1b4305"). InnerVolumeSpecName "encryption-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:28:58.403588 master-1 kubenswrapper[4771]: I1011 10:28:58.403514 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403613 4771 reconciler_common.go:293] "Volume detached for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-encryption-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403630 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403644 4771 reconciler_common.go:293] "Volume detached for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-node-pullsecrets\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403657 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-dbkqb\" (UniqueName: \"kubernetes.io/projected/94d811f4-4ac9-46b0-b937-d3370b1b4305-kube-api-access-dbkqb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403672 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-serving-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403684 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/94d811f4-4ac9-46b0-b937-d3370b1b4305-etcd-client\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403696 4771 reconciler_common.go:293] "Volume detached for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-image-import-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403708 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403720 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.403756 master-1 kubenswrapper[4771]: I1011 10:28:58.403734 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:58.404297 master-1 kubenswrapper[4771]: E1011 10:28:58.403769 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:28:58.404297 master-1 kubenswrapper[4771]: E1011 10:28:58.403885 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:29:02.40385068 +0000 UTC m=+174.378077151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:28:59.167506 master-1 kubenswrapper[4771]: I1011 10:28:59.167462 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-796c687c6d-9b677" Oct 11 10:28:59.198909 master-1 kubenswrapper[4771]: I1011 10:28:59.198842 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-apiserver/apiserver-555f658fd6-n5n6g"] Oct 11 10:28:59.199741 master-1 kubenswrapper[4771]: I1011 10:28:59.199703 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.200301 master-1 kubenswrapper[4771]: I1011 10:28:59.200258 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-apiserver/apiserver-796c687c6d-9b677"] Oct 11 10:28:59.203709 master-1 kubenswrapper[4771]: I1011 10:28:59.203660 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-apiserver/apiserver-796c687c6d-9b677"] Oct 11 10:28:59.203850 master-1 kubenswrapper[4771]: I1011 10:28:59.203824 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"image-import-ca" Oct 11 10:28:59.203931 master-1 kubenswrapper[4771]: I1011 10:28:59.203847 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"etcd-client" Oct 11 10:28:59.203931 master-1 kubenswrapper[4771]: I1011 10:28:59.203853 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"encryption-config-1" Oct 11 10:28:59.204324 master-1 kubenswrapper[4771]: I1011 10:28:59.204293 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"etcd-serving-ca" Oct 11 10:28:59.204324 master-1 kubenswrapper[4771]: I1011 10:28:59.204324 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"config" Oct 11 10:28:59.204547 master-1 kubenswrapper[4771]: I1011 10:28:59.204510 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"audit-1" Oct 11 10:28:59.204779 master-1 kubenswrapper[4771]: I1011 10:28:59.204749 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"serving-cert" Oct 11 10:28:59.204870 master-1 kubenswrapper[4771]: I1011 10:28:59.204847 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"kube-root-ca.crt" Oct 11 10:28:59.205574 master-1 kubenswrapper[4771]: I1011 10:28:59.205532 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"openshift-service-ca.crt" Oct 11 10:28:59.211127 master-1 kubenswrapper[4771]: I1011 10:28:59.211086 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-apiserver/apiserver-555f658fd6-n5n6g"] Oct 11 10:28:59.212296 master-1 kubenswrapper[4771]: I1011 10:28:59.212255 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"trusted-ca-bundle" Oct 11 10:28:59.312269 master-1 kubenswrapper[4771]: I1011 10:28:59.312171 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-client\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312269 master-1 kubenswrapper[4771]: I1011 10:28:59.312268 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-config\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312515 master-1 kubenswrapper[4771]: I1011 10:28:59.312391 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-encryption-config\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312515 master-1 kubenswrapper[4771]: I1011 10:28:59.312429 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-serving-cert\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312515 master-1 kubenswrapper[4771]: I1011 10:28:59.312461 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-serving-ca\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312630 master-1 kubenswrapper[4771]: I1011 10:28:59.312588 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-image-import-ca\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312728 master-1 kubenswrapper[4771]: I1011 10:28:59.312701 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-trusted-ca-bundle\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312772 master-1 kubenswrapper[4771]: I1011 10:28:59.312732 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312804 master-1 kubenswrapper[4771]: I1011 10:28:59.312797 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-node-pullsecrets\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312840 master-1 kubenswrapper[4771]: I1011 10:28:59.312829 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s269\" (UniqueName: \"kubernetes.io/projected/027736d1-f3d3-490e-9ee1-d08bad7a25b7-kube-api-access-4s269\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312882 master-1 kubenswrapper[4771]: I1011 10:28:59.312866 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit-dir\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.312929 master-1 kubenswrapper[4771]: I1011 10:28:59.312918 4771 reconciler_common.go:293] "Volume detached for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/94d811f4-4ac9-46b0-b937-d3370b1b4305-audit\") on node \"master-1\" DevicePath \"\"" Oct 11 10:28:59.414195 master-1 kubenswrapper[4771]: I1011 10:28:59.414030 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-node-pullsecrets\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414195 master-1 kubenswrapper[4771]: I1011 10:28:59.414088 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-4s269\" (UniqueName: \"kubernetes.io/projected/027736d1-f3d3-490e-9ee1-d08bad7a25b7-kube-api-access-4s269\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414195 master-1 kubenswrapper[4771]: I1011 10:28:59.414123 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit-dir\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414195 master-1 kubenswrapper[4771]: I1011 10:28:59.414154 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-client\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414195 master-1 kubenswrapper[4771]: I1011 10:28:59.414177 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-config\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414531 master-1 kubenswrapper[4771]: I1011 10:28:59.414209 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit-dir\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414531 master-1 kubenswrapper[4771]: I1011 10:28:59.414221 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-encryption-config\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414531 master-1 kubenswrapper[4771]: I1011 10:28:59.414225 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-node-pullsecrets\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414531 master-1 kubenswrapper[4771]: I1011 10:28:59.414241 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-serving-cert\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414531 master-1 kubenswrapper[4771]: I1011 10:28:59.414449 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-serving-ca\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.414531 master-1 kubenswrapper[4771]: I1011 10:28:59.414524 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-image-import-ca\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.415209 master-1 kubenswrapper[4771]: I1011 10:28:59.414600 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-trusted-ca-bundle\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.415209 master-1 kubenswrapper[4771]: I1011 10:28:59.414634 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.415209 master-1 kubenswrapper[4771]: I1011 10:28:59.414971 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-config\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.415584 master-1 kubenswrapper[4771]: I1011 10:28:59.415517 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-image-import-ca\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.416021 master-1 kubenswrapper[4771]: I1011 10:28:59.415975 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-trusted-ca-bundle\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.416465 master-1 kubenswrapper[4771]: I1011 10:28:59.416405 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.416572 master-1 kubenswrapper[4771]: I1011 10:28:59.416408 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-serving-ca\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.420072 master-1 kubenswrapper[4771]: I1011 10:28:59.420009 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-serving-cert\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.420269 master-1 kubenswrapper[4771]: I1011 10:28:59.420201 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-encryption-config\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.421165 master-1 kubenswrapper[4771]: I1011 10:28:59.421096 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-client\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.442321 master-1 kubenswrapper[4771]: I1011 10:28:59.441926 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s269\" (UniqueName: \"kubernetes.io/projected/027736d1-f3d3-490e-9ee1-d08bad7a25b7-kube-api-access-4s269\") pod \"apiserver-555f658fd6-n5n6g\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.516217 master-1 kubenswrapper[4771]: I1011 10:28:59.516149 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:28:59.995985 master-1 kubenswrapper[4771]: I1011 10:28:59.995809 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-apiserver/apiserver-555f658fd6-n5n6g"] Oct 11 10:29:00.005573 master-1 kubenswrapper[4771]: W1011 10:29:00.005503 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027736d1_f3d3_490e_9ee1_d08bad7a25b7.slice/crio-0245a7fd6940eab125c14495c22d9aa4a273c8034b951fafcde945d3497b7a29 WatchSource:0}: Error finding container 0245a7fd6940eab125c14495c22d9aa4a273c8034b951fafcde945d3497b7a29: Status 404 returned error can't find the container with id 0245a7fd6940eab125c14495c22d9aa4a273c8034b951fafcde945d3497b7a29 Oct 11 10:29:00.022499 master-1 kubenswrapper[4771]: I1011 10:29:00.022434 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:29:00.022716 master-1 kubenswrapper[4771]: E1011 10:29:00.022635 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:29:00.022838 master-1 kubenswrapper[4771]: E1011 10:29:00.022805 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:29:08.022764781 +0000 UTC m=+179.996991262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:29:00.162111 master-1 kubenswrapper[4771]: I1011 10:29:00.162034 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/installer-1-master-1"] Oct 11 10:29:00.162970 master-1 kubenswrapper[4771]: I1011 10:29:00.162918 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.166432 master-1 kubenswrapper[4771]: I1011 10:29:00.166347 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-controller-manager"/"kube-root-ca.crt" Oct 11 10:29:00.170025 master-1 kubenswrapper[4771]: I1011 10:29:00.169944 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-1-master-1"] Oct 11 10:29:00.172909 master-1 kubenswrapper[4771]: I1011 10:29:00.172840 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" event={"ID":"027736d1-f3d3-490e-9ee1-d08bad7a25b7","Type":"ContainerStarted","Data":"0245a7fd6940eab125c14495c22d9aa4a273c8034b951fafcde945d3497b7a29"} Oct 11 10:29:00.176114 master-1 kubenswrapper[4771]: I1011 10:29:00.176049 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" event={"ID":"b763cbe4-f035-45f2-9f70-4bbb8d5cac87","Type":"ContainerStarted","Data":"9cc8b420bb3cfff536ac575cf0f6beae66dfd3d8bbf199b252ad774b529ae70b"} Oct 11 10:29:00.176114 master-1 kubenswrapper[4771]: I1011 10:29:00.176098 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" event={"ID":"b763cbe4-f035-45f2-9f70-4bbb8d5cac87","Type":"ContainerStarted","Data":"b2ffb3a26e0ef89ca4434fa7ed83e3ba7017b0a577bbe8882a5dee941aa1a52d"} Oct 11 10:29:00.197391 master-1 kubenswrapper[4771]: I1011 10:29:00.197283 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-75f9c7d795-2zgv4" podStartSLOduration=2.185435645 podStartE2EDuration="4.197262832s" podCreationTimestamp="2025-10-11 10:28:56 +0000 UTC" firstStartedPulling="2025-10-11 10:28:57.314660307 +0000 UTC m=+169.288886778" lastFinishedPulling="2025-10-11 10:28:59.326487524 +0000 UTC m=+171.300713965" observedRunningTime="2025-10-11 10:29:00.195517595 +0000 UTC m=+172.169744076" watchObservedRunningTime="2025-10-11 10:29:00.197262832 +0000 UTC m=+172.171489303" Oct 11 10:29:00.327116 master-1 kubenswrapper[4771]: I1011 10:29:00.326997 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-var-lock\") pod \"installer-1-master-1\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.327507 master-1 kubenswrapper[4771]: I1011 10:29:00.327149 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/6534d9db-a553-4c39-bf4a-014a359ee336-kube-api-access\") pod \"installer-1-master-1\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.327507 master-1 kubenswrapper[4771]: I1011 10:29:00.327236 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.429095 master-1 kubenswrapper[4771]: I1011 10:29:00.429013 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/6534d9db-a553-4c39-bf4a-014a359ee336-kube-api-access\") pod \"installer-1-master-1\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.429309 master-1 kubenswrapper[4771]: I1011 10:29:00.429114 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.429309 master-1 kubenswrapper[4771]: I1011 10:29:00.429207 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-var-lock\") pod \"installer-1-master-1\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.429309 master-1 kubenswrapper[4771]: I1011 10:29:00.429299 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-var-lock\") pod \"installer-1-master-1\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.429587 master-1 kubenswrapper[4771]: I1011 10:29:00.429320 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.445067 master-1 kubenswrapper[4771]: I1011 10:29:00.444960 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d811f4-4ac9-46b0-b937-d3370b1b4305" path="/var/lib/kubelet/pods/94d811f4-4ac9-46b0-b937-d3370b1b4305/volumes" Oct 11 10:29:00.462002 master-1 kubenswrapper[4771]: I1011 10:29:00.461918 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/6534d9db-a553-4c39-bf4a-014a359ee336-kube-api-access\") pod \"installer-1-master-1\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.482855 master-1 kubenswrapper[4771]: I1011 10:29:00.482765 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:00.945475 master-1 kubenswrapper[4771]: I1011 10:29:00.945377 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-1-master-1"] Oct 11 10:29:01.138919 master-1 kubenswrapper[4771]: I1011 10:29:01.138830 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:29:01.139214 master-1 kubenswrapper[4771]: E1011 10:29:01.138994 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:29:17.138959671 +0000 UTC m=+189.113186152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:29:01.180238 master-1 kubenswrapper[4771]: I1011 10:29:01.180129 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-1-master-1" event={"ID":"6534d9db-a553-4c39-bf4a-014a359ee336","Type":"ContainerStarted","Data":"b5b289645c8dafc708db0dfb37bf1e6882fdc062aac0a46f6f992e36cadc5dc7"} Oct 11 10:29:01.240296 master-1 kubenswrapper[4771]: I1011 10:29:01.240135 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:29:01.240568 master-1 kubenswrapper[4771]: E1011 10:29:01.240392 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:29:17.24034625 +0000 UTC m=+189.214572741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:29:02.092943 master-1 kubenswrapper[4771]: I1011 10:29:02.092895 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-scheduler/installer-2-master-1"] Oct 11 10:29:02.093233 master-1 kubenswrapper[4771]: I1011 10:29:02.093116 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-scheduler/installer-2-master-1" podUID="750e7efe-07f7-4280-85c7-78250178f965" containerName="installer" containerID="cri-o://2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2" gracePeriod=30 Oct 11 10:29:02.453006 master-1 kubenswrapper[4771]: I1011 10:29:02.452918 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:29:02.454030 master-1 kubenswrapper[4771]: E1011 10:29:02.453046 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:29:02.454030 master-1 kubenswrapper[4771]: E1011 10:29:02.453107 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:29:10.453090763 +0000 UTC m=+182.427317304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:29:02.972720 master-1 kubenswrapper[4771]: I1011 10:29:02.972573 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_installer-2-master-1_750e7efe-07f7-4280-85c7-78250178f965/installer/0.log" Oct 11 10:29:02.972720 master-1 kubenswrapper[4771]: I1011 10:29:02.972649 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:29:03.059646 master-1 kubenswrapper[4771]: I1011 10:29:03.059574 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-var-lock\") pod \"750e7efe-07f7-4280-85c7-78250178f965\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " Oct 11 10:29:03.059773 master-1 kubenswrapper[4771]: I1011 10:29:03.059727 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/750e7efe-07f7-4280-85c7-78250178f965-kube-api-access\") pod \"750e7efe-07f7-4280-85c7-78250178f965\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " Oct 11 10:29:03.059773 master-1 kubenswrapper[4771]: I1011 10:29:03.059736 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-var-lock" (OuterVolumeSpecName: "var-lock") pod "750e7efe-07f7-4280-85c7-78250178f965" (UID: "750e7efe-07f7-4280-85c7-78250178f965"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:03.059841 master-1 kubenswrapper[4771]: I1011 10:29:03.059798 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-kubelet-dir\") pod \"750e7efe-07f7-4280-85c7-78250178f965\" (UID: \"750e7efe-07f7-4280-85c7-78250178f965\") " Oct 11 10:29:03.059934 master-1 kubenswrapper[4771]: I1011 10:29:03.059902 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "750e7efe-07f7-4280-85c7-78250178f965" (UID: "750e7efe-07f7-4280-85c7-78250178f965"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:03.060234 master-1 kubenswrapper[4771]: I1011 10:29:03.060206 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:03.060270 master-1 kubenswrapper[4771]: I1011 10:29:03.060241 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/750e7efe-07f7-4280-85c7-78250178f965-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:03.066601 master-1 kubenswrapper[4771]: I1011 10:29:03.066556 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750e7efe-07f7-4280-85c7-78250178f965-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "750e7efe-07f7-4280-85c7-78250178f965" (UID: "750e7efe-07f7-4280-85c7-78250178f965"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:29:03.161611 master-1 kubenswrapper[4771]: I1011 10:29:03.161564 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/750e7efe-07f7-4280-85c7-78250178f965-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:03.188267 master-1 kubenswrapper[4771]: I1011 10:29:03.188224 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_installer-2-master-1_750e7efe-07f7-4280-85c7-78250178f965/installer/0.log" Oct 11 10:29:03.188393 master-1 kubenswrapper[4771]: I1011 10:29:03.188293 4771 generic.go:334] "Generic (PLEG): container finished" podID="750e7efe-07f7-4280-85c7-78250178f965" containerID="2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2" exitCode=1 Oct 11 10:29:03.188393 master-1 kubenswrapper[4771]: I1011 10:29:03.188330 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-2-master-1" event={"ID":"750e7efe-07f7-4280-85c7-78250178f965","Type":"ContainerDied","Data":"2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2"} Oct 11 10:29:03.188476 master-1 kubenswrapper[4771]: I1011 10:29:03.188408 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-2-master-1" event={"ID":"750e7efe-07f7-4280-85c7-78250178f965","Type":"ContainerDied","Data":"23a2788a32f3ff82c23ee63b5bf39839725cd21b8f03a9a69d479fa53764ca28"} Oct 11 10:29:03.188476 master-1 kubenswrapper[4771]: I1011 10:29:03.188454 4771 scope.go:117] "RemoveContainer" containerID="2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2" Oct 11 10:29:03.188531 master-1 kubenswrapper[4771]: I1011 10:29:03.188468 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-2-master-1" Oct 11 10:29:03.201175 master-1 kubenswrapper[4771]: I1011 10:29:03.201142 4771 scope.go:117] "RemoveContainer" containerID="2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2" Oct 11 10:29:03.201666 master-1 kubenswrapper[4771]: E1011 10:29:03.201605 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2\": container with ID starting with 2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2 not found: ID does not exist" containerID="2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2" Oct 11 10:29:03.201666 master-1 kubenswrapper[4771]: I1011 10:29:03.201639 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2"} err="failed to get container status \"2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2\": rpc error: code = NotFound desc = could not find container \"2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2\": container with ID starting with 2bb08e74f6a03f94e752e699929c9aed89e09c0d951507eccf30a241344f6bd2 not found: ID does not exist" Oct 11 10:29:03.219106 master-1 kubenswrapper[4771]: I1011 10:29:03.219070 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-scheduler/installer-2-master-1"] Oct 11 10:29:03.223398 master-1 kubenswrapper[4771]: I1011 10:29:03.223346 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-scheduler/installer-2-master-1"] Oct 11 10:29:03.436923 master-1 kubenswrapper[4771]: I1011 10:29:03.436821 4771 scope.go:117] "RemoveContainer" containerID="793c72629ffb5d64763cce906980f11774530f02d707e0389b69155b33560c5d" Oct 11 10:29:03.809864 master-1 kubenswrapper[4771]: I1011 10:29:03.809778 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rzjcf" Oct 11 10:29:04.197756 master-1 kubenswrapper[4771]: I1011 10:29:04.197617 4771 generic.go:334] "Generic (PLEG): container finished" podID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerID="41af63c058a1e7b90357082e0adac794e0e1b2996f71cfa6b9c3a91b7079c8d7" exitCode=0 Oct 11 10:29:04.198097 master-1 kubenswrapper[4771]: I1011 10:29:04.197702 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" event={"ID":"027736d1-f3d3-490e-9ee1-d08bad7a25b7","Type":"ContainerDied","Data":"41af63c058a1e7b90357082e0adac794e0e1b2996f71cfa6b9c3a91b7079c8d7"} Oct 11 10:29:04.202081 master-1 kubenswrapper[4771]: I1011 10:29:04.201982 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-1-master-1" event={"ID":"6534d9db-a553-4c39-bf4a-014a359ee336","Type":"ContainerStarted","Data":"c9e465db2f016eeb1b9eb6a1701316ad91386e0556613224875082e886221894"} Oct 11 10:29:04.213999 master-1 kubenswrapper[4771]: I1011 10:29:04.213941 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/3.log" Oct 11 10:29:04.216030 master-1 kubenswrapper[4771]: I1011 10:29:04.215953 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager-operator/cluster-cloud-controller-manager-operator-779749f859-5xxzp" event={"ID":"e115f8be-9e65-4407-8111-568e5ea8ac1b","Type":"ContainerStarted","Data":"903a925fbd464397f1aac6d43f29ca7e35957aff84f8e3ba36189e56cf222199"} Oct 11 10:29:04.255246 master-1 kubenswrapper[4771]: I1011 10:29:04.255124 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/installer-1-master-1" podStartSLOduration=2.228838693 podStartE2EDuration="4.255098543s" podCreationTimestamp="2025-10-11 10:29:00 +0000 UTC" firstStartedPulling="2025-10-11 10:29:00.95580279 +0000 UTC m=+172.930029231" lastFinishedPulling="2025-10-11 10:29:02.98206264 +0000 UTC m=+174.956289081" observedRunningTime="2025-10-11 10:29:04.253290695 +0000 UTC m=+176.227517196" watchObservedRunningTime="2025-10-11 10:29:04.255098543 +0000 UTC m=+176.229325014" Oct 11 10:29:04.446552 master-1 kubenswrapper[4771]: I1011 10:29:04.446458 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="750e7efe-07f7-4280-85c7-78250178f965" path="/var/lib/kubelet/pods/750e7efe-07f7-4280-85c7-78250178f965/volumes" Oct 11 10:29:04.692604 master-1 kubenswrapper[4771]: I1011 10:29:04.692532 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/installer-3-master-1"] Oct 11 10:29:04.692806 master-1 kubenswrapper[4771]: E1011 10:29:04.692725 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="750e7efe-07f7-4280-85c7-78250178f965" containerName="installer" Oct 11 10:29:04.692806 master-1 kubenswrapper[4771]: I1011 10:29:04.692744 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="750e7efe-07f7-4280-85c7-78250178f965" containerName="installer" Oct 11 10:29:04.692866 master-1 kubenswrapper[4771]: I1011 10:29:04.692837 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="750e7efe-07f7-4280-85c7-78250178f965" containerName="installer" Oct 11 10:29:04.693260 master-1 kubenswrapper[4771]: I1011 10:29:04.693233 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:04.700691 master-1 kubenswrapper[4771]: I1011 10:29:04.700602 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-3-master-1"] Oct 11 10:29:04.779388 master-1 kubenswrapper[4771]: I1011 10:29:04.779297 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-kubelet-dir\") pod \"installer-3-master-1\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:04.779656 master-1 kubenswrapper[4771]: I1011 10:29:04.779403 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/49766d35-174a-4677-8b2d-e3ed195d0a26-kube-api-access\") pod \"installer-3-master-1\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:04.779656 master-1 kubenswrapper[4771]: I1011 10:29:04.779475 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-var-lock\") pod \"installer-3-master-1\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:04.880387 master-1 kubenswrapper[4771]: I1011 10:29:04.880307 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-kubelet-dir\") pod \"installer-3-master-1\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:04.880987 master-1 kubenswrapper[4771]: I1011 10:29:04.880434 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/49766d35-174a-4677-8b2d-e3ed195d0a26-kube-api-access\") pod \"installer-3-master-1\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:04.880987 master-1 kubenswrapper[4771]: I1011 10:29:04.880513 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-var-lock\") pod \"installer-3-master-1\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:04.880987 master-1 kubenswrapper[4771]: I1011 10:29:04.880537 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-kubelet-dir\") pod \"installer-3-master-1\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:04.880987 master-1 kubenswrapper[4771]: I1011 10:29:04.880599 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-var-lock\") pod \"installer-3-master-1\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:04.904500 master-1 kubenswrapper[4771]: I1011 10:29:04.903991 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/49766d35-174a-4677-8b2d-e3ed195d0a26-kube-api-access\") pod \"installer-3-master-1\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:05.051693 master-1 kubenswrapper[4771]: I1011 10:29:05.051174 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:05.227388 master-1 kubenswrapper[4771]: I1011 10:29:05.221464 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" event={"ID":"027736d1-f3d3-490e-9ee1-d08bad7a25b7","Type":"ContainerStarted","Data":"5ee744232b5a66fa90e18d0677b90fd7ff50cae1f9e1afc9158b036b712f32da"} Oct 11 10:29:05.247734 master-1 kubenswrapper[4771]: I1011 10:29:05.247676 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-3-master-1"] Oct 11 10:29:05.253618 master-1 kubenswrapper[4771]: W1011 10:29:05.253560 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49766d35_174a_4677_8b2d_e3ed195d0a26.slice/crio-c590919bb57c8eb20558e4148e5af322c480e4ab020811da46e7b9c3247ad430 WatchSource:0}: Error finding container c590919bb57c8eb20558e4148e5af322c480e4ab020811da46e7b9c3247ad430: Status 404 returned error can't find the container with id c590919bb57c8eb20558e4148e5af322c480e4ab020811da46e7b9c3247ad430 Oct 11 10:29:06.226242 master-1 kubenswrapper[4771]: I1011 10:29:06.226164 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-3-master-1" event={"ID":"49766d35-174a-4677-8b2d-e3ed195d0a26","Type":"ContainerStarted","Data":"7f8db2473bcbc14ad35cb8dd456f940c8050a9882fcf9aa519950777d8bb0fc0"} Oct 11 10:29:06.226242 master-1 kubenswrapper[4771]: I1011 10:29:06.226218 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-3-master-1" event={"ID":"49766d35-174a-4677-8b2d-e3ed195d0a26","Type":"ContainerStarted","Data":"c590919bb57c8eb20558e4148e5af322c480e4ab020811da46e7b9c3247ad430"} Oct 11 10:29:07.233736 master-1 kubenswrapper[4771]: I1011 10:29:07.233526 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" event={"ID":"027736d1-f3d3-490e-9ee1-d08bad7a25b7","Type":"ContainerStarted","Data":"893d86a98f61447fa7f11deae879fe95aeccf34e5a1d5e59961a43c4a181ec43"} Oct 11 10:29:07.261146 master-1 kubenswrapper[4771]: I1011 10:29:07.261029 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/installer-3-master-1" podStartSLOduration=3.261005871 podStartE2EDuration="3.261005871s" podCreationTimestamp="2025-10-11 10:29:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:06.246433927 +0000 UTC m=+178.220660378" watchObservedRunningTime="2025-10-11 10:29:07.261005871 +0000 UTC m=+179.235232352" Oct 11 10:29:07.262296 master-1 kubenswrapper[4771]: I1011 10:29:07.262229 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podStartSLOduration=2.610306823 podStartE2EDuration="9.262221854s" podCreationTimestamp="2025-10-11 10:28:58 +0000 UTC" firstStartedPulling="2025-10-11 10:29:00.009722205 +0000 UTC m=+171.983948676" lastFinishedPulling="2025-10-11 10:29:06.661637226 +0000 UTC m=+178.635863707" observedRunningTime="2025-10-11 10:29:07.25982775 +0000 UTC m=+179.234054251" watchObservedRunningTime="2025-10-11 10:29:07.262221854 +0000 UTC m=+179.236448325" Oct 11 10:29:08.120313 master-1 kubenswrapper[4771]: I1011 10:29:08.120225 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:29:08.120647 master-1 kubenswrapper[4771]: E1011 10:29:08.120528 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:29:08.120683 master-1 kubenswrapper[4771]: E1011 10:29:08.120661 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:29:24.120633123 +0000 UTC m=+196.094859594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:29:09.517085 master-1 kubenswrapper[4771]: I1011 10:29:09.516973 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:29:09.517085 master-1 kubenswrapper[4771]: I1011 10:29:09.517071 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:29:09.527819 master-1 kubenswrapper[4771]: I1011 10:29:09.527761 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:29:10.254840 master-1 kubenswrapper[4771]: I1011 10:29:10.254755 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:29:10.547686 master-1 kubenswrapper[4771]: I1011 10:29:10.547534 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:29:10.548478 master-1 kubenswrapper[4771]: E1011 10:29:10.547698 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:29:10.548478 master-1 kubenswrapper[4771]: E1011 10:29:10.547806 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:29:26.547775572 +0000 UTC m=+198.522002043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:29:12.092532 master-1 kubenswrapper[4771]: I1011 10:29:12.092436 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-scheduler/installer-3-master-1"] Oct 11 10:29:12.093550 master-1 kubenswrapper[4771]: I1011 10:29:12.092682 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-scheduler/installer-3-master-1" podUID="49766d35-174a-4677-8b2d-e3ed195d0a26" containerName="installer" containerID="cri-o://7f8db2473bcbc14ad35cb8dd456f940c8050a9882fcf9aa519950777d8bb0fc0" gracePeriod=30 Oct 11 10:29:12.272423 master-1 kubenswrapper[4771]: I1011 10:29:12.272323 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_installer-3-master-1_49766d35-174a-4677-8b2d-e3ed195d0a26/installer/0.log" Oct 11 10:29:12.272687 master-1 kubenswrapper[4771]: I1011 10:29:12.272431 4771 generic.go:334] "Generic (PLEG): container finished" podID="49766d35-174a-4677-8b2d-e3ed195d0a26" containerID="7f8db2473bcbc14ad35cb8dd456f940c8050a9882fcf9aa519950777d8bb0fc0" exitCode=1 Oct 11 10:29:12.272687 master-1 kubenswrapper[4771]: I1011 10:29:12.272488 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-3-master-1" event={"ID":"49766d35-174a-4677-8b2d-e3ed195d0a26","Type":"ContainerDied","Data":"7f8db2473bcbc14ad35cb8dd456f940c8050a9882fcf9aa519950777d8bb0fc0"} Oct 11 10:29:12.436134 master-1 kubenswrapper[4771]: I1011 10:29:12.435983 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/machine-config-daemon-9nzpz"] Oct 11 10:29:12.437118 master-1 kubenswrapper[4771]: I1011 10:29:12.437068 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.442257 master-1 kubenswrapper[4771]: I1011 10:29:12.442203 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-machine-config-operator"/"proxy-tls" Oct 11 10:29:12.442799 master-1 kubenswrapper[4771]: I1011 10:29:12.442769 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-machine-config-operator"/"kube-rbac-proxy" Oct 11 10:29:12.442922 master-1 kubenswrapper[4771]: I1011 10:29:12.442883 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-machine-config-operator"/"openshift-service-ca.crt" Oct 11 10:29:12.443099 master-1 kubenswrapper[4771]: I1011 10:29:12.443064 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-machine-config-operator"/"kube-root-ca.crt" Oct 11 10:29:12.501883 master-1 kubenswrapper[4771]: I1011 10:29:12.501829 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_installer-3-master-1_49766d35-174a-4677-8b2d-e3ed195d0a26/installer/0.log" Oct 11 10:29:12.502012 master-1 kubenswrapper[4771]: I1011 10:29:12.501908 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:12.569486 master-1 kubenswrapper[4771]: I1011 10:29:12.569410 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"mcd-auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-mcd-auth-proxy-config\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.569486 master-1 kubenswrapper[4771]: I1011 10:29:12.569456 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-proxy-tls\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.569486 master-1 kubenswrapper[4771]: I1011 10:29:12.569496 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rootfs\" (UniqueName: \"kubernetes.io/host-path/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-rootfs\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.569879 master-1 kubenswrapper[4771]: I1011 10:29:12.569516 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzt9h\" (UniqueName: \"kubernetes.io/projected/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-kube-api-access-kzt9h\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.670191 master-1 kubenswrapper[4771]: I1011 10:29:12.670059 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-var-lock\") pod \"49766d35-174a-4677-8b2d-e3ed195d0a26\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " Oct 11 10:29:12.670191 master-1 kubenswrapper[4771]: I1011 10:29:12.670186 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-var-lock" (OuterVolumeSpecName: "var-lock") pod "49766d35-174a-4677-8b2d-e3ed195d0a26" (UID: "49766d35-174a-4677-8b2d-e3ed195d0a26"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:12.670552 master-1 kubenswrapper[4771]: I1011 10:29:12.670212 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-kubelet-dir\") pod \"49766d35-174a-4677-8b2d-e3ed195d0a26\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " Oct 11 10:29:12.670552 master-1 kubenswrapper[4771]: I1011 10:29:12.670238 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "49766d35-174a-4677-8b2d-e3ed195d0a26" (UID: "49766d35-174a-4677-8b2d-e3ed195d0a26"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:12.670552 master-1 kubenswrapper[4771]: I1011 10:29:12.670322 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/49766d35-174a-4677-8b2d-e3ed195d0a26-kube-api-access\") pod \"49766d35-174a-4677-8b2d-e3ed195d0a26\" (UID: \"49766d35-174a-4677-8b2d-e3ed195d0a26\") " Oct 11 10:29:12.670800 master-1 kubenswrapper[4771]: I1011 10:29:12.670608 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt9h\" (UniqueName: \"kubernetes.io/projected/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-kube-api-access-kzt9h\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.670897 master-1 kubenswrapper[4771]: I1011 10:29:12.670875 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"mcd-auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-mcd-auth-proxy-config\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.670988 master-1 kubenswrapper[4771]: I1011 10:29:12.670916 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-proxy-tls\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.670988 master-1 kubenswrapper[4771]: I1011 10:29:12.670972 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"rootfs\" (UniqueName: \"kubernetes.io/host-path/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-rootfs\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.671205 master-1 kubenswrapper[4771]: I1011 10:29:12.671074 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:12.671205 master-1 kubenswrapper[4771]: I1011 10:29:12.671101 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49766d35-174a-4677-8b2d-e3ed195d0a26-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:12.671205 master-1 kubenswrapper[4771]: I1011 10:29:12.671192 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"rootfs\" (UniqueName: \"kubernetes.io/host-path/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-rootfs\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.672102 master-1 kubenswrapper[4771]: I1011 10:29:12.672028 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"mcd-auth-proxy-config\" (UniqueName: \"kubernetes.io/configmap/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-mcd-auth-proxy-config\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.674835 master-1 kubenswrapper[4771]: I1011 10:29:12.674765 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49766d35-174a-4677-8b2d-e3ed195d0a26-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "49766d35-174a-4677-8b2d-e3ed195d0a26" (UID: "49766d35-174a-4677-8b2d-e3ed195d0a26"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:29:12.675764 master-1 kubenswrapper[4771]: I1011 10:29:12.675719 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-proxy-tls\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.694842 master-1 kubenswrapper[4771]: I1011 10:29:12.694636 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzt9h\" (UniqueName: \"kubernetes.io/projected/ebb73d72-cbb7-4736-870e-79e86c9fa7f5-kube-api-access-kzt9h\") pod \"machine-config-daemon-9nzpz\" (UID: \"ebb73d72-cbb7-4736-870e-79e86c9fa7f5\") " pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.756272 master-1 kubenswrapper[4771]: I1011 10:29:12.756124 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" Oct 11 10:29:12.771940 master-1 kubenswrapper[4771]: I1011 10:29:12.771870 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/49766d35-174a-4677-8b2d-e3ed195d0a26-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:12.883333 master-1 kubenswrapper[4771]: I1011 10:29:12.883225 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4pm7x" Oct 11 10:29:13.277499 master-1 kubenswrapper[4771]: I1011 10:29:13.277460 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_installer-3-master-1_49766d35-174a-4677-8b2d-e3ed195d0a26/installer/0.log" Oct 11 10:29:13.278215 master-1 kubenswrapper[4771]: I1011 10:29:13.277593 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-3-master-1" Oct 11 10:29:13.278847 master-1 kubenswrapper[4771]: I1011 10:29:13.278427 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-3-master-1" event={"ID":"49766d35-174a-4677-8b2d-e3ed195d0a26","Type":"ContainerDied","Data":"c590919bb57c8eb20558e4148e5af322c480e4ab020811da46e7b9c3247ad430"} Oct 11 10:29:13.278847 master-1 kubenswrapper[4771]: I1011 10:29:13.278568 4771 scope.go:117] "RemoveContainer" containerID="7f8db2473bcbc14ad35cb8dd456f940c8050a9882fcf9aa519950777d8bb0fc0" Oct 11 10:29:13.285710 master-1 kubenswrapper[4771]: I1011 10:29:13.284556 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" event={"ID":"ebb73d72-cbb7-4736-870e-79e86c9fa7f5","Type":"ContainerStarted","Data":"101752d726c211a4147854bc821564d59ba9692b10a61e8a2aebecd7573d5028"} Oct 11 10:29:13.285710 master-1 kubenswrapper[4771]: I1011 10:29:13.284614 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" event={"ID":"ebb73d72-cbb7-4736-870e-79e86c9fa7f5","Type":"ContainerStarted","Data":"bbebfd3c58947a78314370999f26a02214de1f1c409cbccc2d5da4102c7788b3"} Oct 11 10:29:13.285710 master-1 kubenswrapper[4771]: I1011 10:29:13.284640 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" event={"ID":"ebb73d72-cbb7-4736-870e-79e86c9fa7f5","Type":"ContainerStarted","Data":"461a26a3bdce412f46dca35747cd452487253f3acb4cd435bdee26610015c724"} Oct 11 10:29:13.302454 master-1 kubenswrapper[4771]: I1011 10:29:13.302339 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" podStartSLOduration=1.302311408 podStartE2EDuration="1.302311408s" podCreationTimestamp="2025-10-11 10:29:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:13.300503579 +0000 UTC m=+185.274730080" watchObservedRunningTime="2025-10-11 10:29:13.302311408 +0000 UTC m=+185.276537849" Oct 11 10:29:13.345812 master-1 kubenswrapper[4771]: I1011 10:29:13.345722 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-scheduler/installer-3-master-1"] Oct 11 10:29:13.345924 master-1 kubenswrapper[4771]: I1011 10:29:13.345830 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-scheduler/installer-3-master-1"] Oct 11 10:29:14.441608 master-1 kubenswrapper[4771]: I1011 10:29:14.441560 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49766d35-174a-4677-8b2d-e3ed195d0a26" path="/var/lib/kubelet/pods/49766d35-174a-4677-8b2d-e3ed195d0a26/volumes" Oct 11 10:29:14.497896 master-1 kubenswrapper[4771]: I1011 10:29:14.497786 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/installer-4-master-1"] Oct 11 10:29:14.498209 master-1 kubenswrapper[4771]: E1011 10:29:14.498013 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="49766d35-174a-4677-8b2d-e3ed195d0a26" containerName="installer" Oct 11 10:29:14.498209 master-1 kubenswrapper[4771]: I1011 10:29:14.498039 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="49766d35-174a-4677-8b2d-e3ed195d0a26" containerName="installer" Oct 11 10:29:14.498209 master-1 kubenswrapper[4771]: I1011 10:29:14.498151 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="49766d35-174a-4677-8b2d-e3ed195d0a26" containerName="installer" Oct 11 10:29:14.498759 master-1 kubenswrapper[4771]: I1011 10:29:14.498703 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.506500 master-1 kubenswrapper[4771]: I1011 10:29:14.506442 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-4-master-1"] Oct 11 10:29:14.591925 master-1 kubenswrapper[4771]: I1011 10:29:14.591808 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-kubelet-dir\") pod \"installer-4-master-1\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.591925 master-1 kubenswrapper[4771]: I1011 10:29:14.591888 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-var-lock\") pod \"installer-4-master-1\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.592333 master-1 kubenswrapper[4771]: I1011 10:29:14.591978 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/7662f87a-13ba-439c-b386-05e68284803c-kube-api-access\") pod \"installer-4-master-1\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.592600 master-1 kubenswrapper[4771]: E1011 10:29:14.592521 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-pod49766d35_174a_4677_8b2d_e3ed195d0a26.slice/crio-c590919bb57c8eb20558e4148e5af322c480e4ab020811da46e7b9c3247ad430\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod007dcbab_9e3e_4dcd_9ad9_0ea8dd07dfc7.slice/crio-conmon-80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod49766d35_174a_4677_8b2d_e3ed195d0a26.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod007dcbab_9e3e_4dcd_9ad9_0ea8dd07dfc7.slice/crio-80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:29:14.693444 master-1 kubenswrapper[4771]: I1011 10:29:14.693224 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/7662f87a-13ba-439c-b386-05e68284803c-kube-api-access\") pod \"installer-4-master-1\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.693444 master-1 kubenswrapper[4771]: I1011 10:29:14.693377 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-kubelet-dir\") pod \"installer-4-master-1\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.693444 master-1 kubenswrapper[4771]: I1011 10:29:14.693400 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-var-lock\") pod \"installer-4-master-1\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.693755 master-1 kubenswrapper[4771]: I1011 10:29:14.693517 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-var-lock\") pod \"installer-4-master-1\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.693755 master-1 kubenswrapper[4771]: I1011 10:29:14.693526 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-kubelet-dir\") pod \"installer-4-master-1\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.717819 master-1 kubenswrapper[4771]: I1011 10:29:14.717757 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/7662f87a-13ba-439c-b386-05e68284803c-kube-api-access\") pod \"installer-4-master-1\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.757536 master-1 kubenswrapper[4771]: I1011 10:29:14.757425 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-controller-manager/installer-1-master-1"] Oct 11 10:29:14.757849 master-1 kubenswrapper[4771]: I1011 10:29:14.757768 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/installer-1-master-1" podUID="6534d9db-a553-4c39-bf4a-014a359ee336" containerName="installer" containerID="cri-o://c9e465db2f016eeb1b9eb6a1701316ad91386e0556613224875082e886221894" gracePeriod=30 Oct 11 10:29:14.831688 master-1 kubenswrapper[4771]: I1011 10:29:14.831642 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_installer-1-master-1_007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7/installer/0.log" Oct 11 10:29:14.831798 master-1 kubenswrapper[4771]: I1011 10:29:14.831729 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:29:14.856425 master-1 kubenswrapper[4771]: I1011 10:29:14.853518 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:14.997236 master-1 kubenswrapper[4771]: I1011 10:29:14.997172 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kube-api-access\") pod \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " Oct 11 10:29:14.997471 master-1 kubenswrapper[4771]: I1011 10:29:14.997281 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-var-lock\") pod \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " Oct 11 10:29:14.997471 master-1 kubenswrapper[4771]: I1011 10:29:14.997311 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kubelet-dir\") pod \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\" (UID: \"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7\") " Oct 11 10:29:14.997562 master-1 kubenswrapper[4771]: I1011 10:29:14.997447 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-var-lock" (OuterVolumeSpecName: "var-lock") pod "007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7" (UID: "007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:14.997562 master-1 kubenswrapper[4771]: I1011 10:29:14.997509 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7" (UID: "007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:15.003160 master-1 kubenswrapper[4771]: I1011 10:29:15.003094 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7" (UID: "007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:29:15.099164 master-1 kubenswrapper[4771]: I1011 10:29:15.099018 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:15.099164 master-1 kubenswrapper[4771]: I1011 10:29:15.099119 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:15.099164 master-1 kubenswrapper[4771]: I1011 10:29:15.099143 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:15.278829 master-1 kubenswrapper[4771]: I1011 10:29:15.278639 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-4-master-1"] Oct 11 10:29:15.287461 master-1 kubenswrapper[4771]: W1011 10:29:15.287347 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7662f87a_13ba_439c_b386_05e68284803c.slice/crio-ed6acc7afd35b0ef55fa3ef023c51664249170487a6297da51f4f2e72955fbfc WatchSource:0}: Error finding container ed6acc7afd35b0ef55fa3ef023c51664249170487a6297da51f4f2e72955fbfc: Status 404 returned error can't find the container with id ed6acc7afd35b0ef55fa3ef023c51664249170487a6297da51f4f2e72955fbfc Oct 11 10:29:15.294300 master-1 kubenswrapper[4771]: I1011 10:29:15.294255 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-4-master-1" event={"ID":"7662f87a-13ba-439c-b386-05e68284803c","Type":"ContainerStarted","Data":"ed6acc7afd35b0ef55fa3ef023c51664249170487a6297da51f4f2e72955fbfc"} Oct 11 10:29:15.296004 master-1 kubenswrapper[4771]: I1011 10:29:15.295941 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_installer-1-master-1_007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7/installer/0.log" Oct 11 10:29:15.296082 master-1 kubenswrapper[4771]: I1011 10:29:15.296012 4771 generic.go:334] "Generic (PLEG): container finished" podID="007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7" containerID="80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90" exitCode=1 Oct 11 10:29:15.296082 master-1 kubenswrapper[4771]: I1011 10:29:15.296057 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-1-master-1" event={"ID":"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7","Type":"ContainerDied","Data":"80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90"} Oct 11 10:29:15.296169 master-1 kubenswrapper[4771]: I1011 10:29:15.296091 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-1-master-1" event={"ID":"007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7","Type":"ContainerDied","Data":"862c7a0762d99806cbf395198a8a115efc48c49b1a93e1cd22e9f82545990f2e"} Oct 11 10:29:15.296169 master-1 kubenswrapper[4771]: I1011 10:29:15.296116 4771 scope.go:117] "RemoveContainer" containerID="80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90" Oct 11 10:29:15.296290 master-1 kubenswrapper[4771]: I1011 10:29:15.296250 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-1-master-1" Oct 11 10:29:15.322948 master-1 kubenswrapper[4771]: I1011 10:29:15.322885 4771 scope.go:117] "RemoveContainer" containerID="80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90" Oct 11 10:29:15.323648 master-1 kubenswrapper[4771]: E1011 10:29:15.323587 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90\": container with ID starting with 80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90 not found: ID does not exist" containerID="80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90" Oct 11 10:29:15.323717 master-1 kubenswrapper[4771]: I1011 10:29:15.323661 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90"} err="failed to get container status \"80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90\": rpc error: code = NotFound desc = could not find container \"80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90\": container with ID starting with 80fbdcaea7022dfed31b23d1e5cd04123cb13507148681a1c855ca79f442ec90 not found: ID does not exist" Oct 11 10:29:15.334271 master-1 kubenswrapper[4771]: I1011 10:29:15.334217 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-scheduler/installer-1-master-1"] Oct 11 10:29:15.337812 master-1 kubenswrapper[4771]: I1011 10:29:15.337766 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-scheduler/installer-1-master-1"] Oct 11 10:29:16.303213 master-1 kubenswrapper[4771]: I1011 10:29:16.303094 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-4-master-1" event={"ID":"7662f87a-13ba-439c-b386-05e68284803c","Type":"ContainerStarted","Data":"6597ee1a813020ee9e9d9c3bc4ac9547370cdcefee548bc443d67590ef76026d"} Oct 11 10:29:16.444524 master-1 kubenswrapper[4771]: I1011 10:29:16.444437 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7" path="/var/lib/kubelet/pods/007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7/volumes" Oct 11 10:29:17.230010 master-1 kubenswrapper[4771]: I1011 10:29:17.229898 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:29:17.230301 master-1 kubenswrapper[4771]: E1011 10:29:17.230180 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:29:49.230137499 +0000 UTC m=+221.204363980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:29:17.330730 master-1 kubenswrapper[4771]: I1011 10:29:17.330635 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:29:17.331618 master-1 kubenswrapper[4771]: E1011 10:29:17.330891 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:29:49.330861222 +0000 UTC m=+221.305087663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:29:17.556131 master-1 kubenswrapper[4771]: I1011 10:29:17.556006 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/installer-4-master-1" podStartSLOduration=3.555982437 podStartE2EDuration="3.555982437s" podCreationTimestamp="2025-10-11 10:29:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:16.319624578 +0000 UTC m=+188.293851119" watchObservedRunningTime="2025-10-11 10:29:17.555982437 +0000 UTC m=+189.530208878" Oct 11 10:29:17.556753 master-1 kubenswrapper[4771]: I1011 10:29:17.556707 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/installer-2-master-1"] Oct 11 10:29:17.556877 master-1 kubenswrapper[4771]: E1011 10:29:17.556850 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7" containerName="installer" Oct 11 10:29:17.556877 master-1 kubenswrapper[4771]: I1011 10:29:17.556864 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7" containerName="installer" Oct 11 10:29:17.557067 master-1 kubenswrapper[4771]: I1011 10:29:17.556924 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="007dcbab-9e3e-4dcd-9ad9-0ea8dd07dfc7" containerName="installer" Oct 11 10:29:17.557266 master-1 kubenswrapper[4771]: I1011 10:29:17.557222 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.567800 master-1 kubenswrapper[4771]: I1011 10:29:17.567728 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-2-master-1"] Oct 11 10:29:17.735184 master-1 kubenswrapper[4771]: I1011 10:29:17.735075 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-var-lock\") pod \"installer-2-master-1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.735519 master-1 kubenswrapper[4771]: I1011 10:29:17.735400 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kubelet-dir\") pod \"installer-2-master-1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.735519 master-1 kubenswrapper[4771]: I1011 10:29:17.735483 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kube-api-access\") pod \"installer-2-master-1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.837215 master-1 kubenswrapper[4771]: I1011 10:29:17.837011 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kubelet-dir\") pod \"installer-2-master-1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.837215 master-1 kubenswrapper[4771]: I1011 10:29:17.837109 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kube-api-access\") pod \"installer-2-master-1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.837215 master-1 kubenswrapper[4771]: I1011 10:29:17.837183 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-var-lock\") pod \"installer-2-master-1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.837649 master-1 kubenswrapper[4771]: I1011 10:29:17.837220 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kubelet-dir\") pod \"installer-2-master-1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.837649 master-1 kubenswrapper[4771]: I1011 10:29:17.837339 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-var-lock\") pod \"installer-2-master-1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.869203 master-1 kubenswrapper[4771]: I1011 10:29:17.869115 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kube-api-access\") pod \"installer-2-master-1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:17.871842 master-1 kubenswrapper[4771]: I1011 10:29:17.871797 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:18.136998 master-1 kubenswrapper[4771]: I1011 10:29:18.136826 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-2-master-1"] Oct 11 10:29:18.145260 master-1 kubenswrapper[4771]: W1011 10:29:18.145198 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod792389a1_400d_4a07_a0a5_e80b2edfd8f1.slice/crio-278d42f198fc93ee50b135376d28ae4eb2fe4bcf6f5f1c9223b4e9e7ffd7be30 WatchSource:0}: Error finding container 278d42f198fc93ee50b135376d28ae4eb2fe4bcf6f5f1c9223b4e9e7ffd7be30: Status 404 returned error can't find the container with id 278d42f198fc93ee50b135376d28ae4eb2fe4bcf6f5f1c9223b4e9e7ffd7be30 Oct 11 10:29:18.315016 master-1 kubenswrapper[4771]: I1011 10:29:18.314535 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-2-master-1" event={"ID":"792389a1-400d-4a07-a0a5-e80b2edfd8f1","Type":"ContainerStarted","Data":"278d42f198fc93ee50b135376d28ae4eb2fe4bcf6f5f1c9223b4e9e7ffd7be30"} Oct 11 10:29:19.322701 master-1 kubenswrapper[4771]: I1011 10:29:19.322613 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-2-master-1" event={"ID":"792389a1-400d-4a07-a0a5-e80b2edfd8f1","Type":"ContainerStarted","Data":"1847b9a9f31d4cf6b7fede3d6231e62c7c7aec1680e7c800a880c6ba363a8798"} Oct 11 10:29:19.342978 master-1 kubenswrapper[4771]: I1011 10:29:19.342824 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/installer-2-master-1" podStartSLOduration=2.342799813 podStartE2EDuration="2.342799813s" podCreationTimestamp="2025-10-11 10:29:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:19.342788622 +0000 UTC m=+191.317015063" watchObservedRunningTime="2025-10-11 10:29:19.342799813 +0000 UTC m=+191.317026274" Oct 11 10:29:21.333886 master-1 kubenswrapper[4771]: I1011 10:29:21.333795 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/community-operators-gwwz9"] Oct 11 10:29:21.334994 master-1 kubenswrapper[4771]: I1011 10:29:21.334814 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.338742 master-1 kubenswrapper[4771]: I1011 10:29:21.338684 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-marketplace"/"openshift-service-ca.crt" Oct 11 10:29:21.338844 master-1 kubenswrapper[4771]: I1011 10:29:21.338791 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-marketplace"/"kube-root-ca.crt" Oct 11 10:29:21.340900 master-1 kubenswrapper[4771]: I1011 10:29:21.340846 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-gwwz9"] Oct 11 10:29:21.478954 master-1 kubenswrapper[4771]: I1011 10:29:21.478814 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-catalog-content\") pod \"community-operators-gwwz9\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.478954 master-1 kubenswrapper[4771]: I1011 10:29:21.478952 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ptz\" (UniqueName: \"kubernetes.io/projected/0b7d1d62-0062-47cd-a963-63893777198e-kube-api-access-r6ptz\") pod \"community-operators-gwwz9\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.479337 master-1 kubenswrapper[4771]: I1011 10:29:21.479078 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-utilities\") pod \"community-operators-gwwz9\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.580488 master-1 kubenswrapper[4771]: I1011 10:29:21.580387 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ptz\" (UniqueName: \"kubernetes.io/projected/0b7d1d62-0062-47cd-a963-63893777198e-kube-api-access-r6ptz\") pod \"community-operators-gwwz9\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.580723 master-1 kubenswrapper[4771]: I1011 10:29:21.580522 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-utilities\") pod \"community-operators-gwwz9\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.580723 master-1 kubenswrapper[4771]: I1011 10:29:21.580588 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-catalog-content\") pod \"community-operators-gwwz9\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.581131 master-1 kubenswrapper[4771]: I1011 10:29:21.581102 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-catalog-content\") pod \"community-operators-gwwz9\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.581499 master-1 kubenswrapper[4771]: I1011 10:29:21.581414 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-utilities\") pod \"community-operators-gwwz9\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.610901 master-1 kubenswrapper[4771]: I1011 10:29:21.610754 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ptz\" (UniqueName: \"kubernetes.io/projected/0b7d1d62-0062-47cd-a963-63893777198e-kube-api-access-r6ptz\") pod \"community-operators-gwwz9\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:21.659761 master-1 kubenswrapper[4771]: I1011 10:29:21.659664 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:22.032844 master-1 kubenswrapper[4771]: I1011 10:29:22.032639 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:29:22.036199 master-1 kubenswrapper[4771]: I1011 10:29:22.036151 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c084572-a5c9-4787-8a14-b7d6b0810a1b-metrics-certs\") pod \"network-metrics-daemon-fgjvw\" (UID: \"2c084572-a5c9-4787-8a14-b7d6b0810a1b\") " pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:29:22.063131 master-1 kubenswrapper[4771]: I1011 10:29:22.063040 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fgjvw" Oct 11 10:29:22.107956 master-1 kubenswrapper[4771]: I1011 10:29:22.107913 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-gwwz9"] Oct 11 10:29:22.179573 master-1 kubenswrapper[4771]: I1011 10:29:22.178739 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-oauth-apiserver_apiserver-65b6f4d4c9-skwvw_004ee387-d0e9-4582-ad14-f571832ebd6e/fix-audit-permissions/0.log" Oct 11 10:29:22.278555 master-1 kubenswrapper[4771]: I1011 10:29:22.278468 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fgjvw"] Oct 11 10:29:22.289927 master-1 kubenswrapper[4771]: W1011 10:29:22.289733 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c084572_a5c9_4787_8a14_b7d6b0810a1b.slice/crio-a3c2aac5746fe934f86d97bbbc78969442131582cbfeb761d1f7e839717b9136 WatchSource:0}: Error finding container a3c2aac5746fe934f86d97bbbc78969442131582cbfeb761d1f7e839717b9136: Status 404 returned error can't find the container with id a3c2aac5746fe934f86d97bbbc78969442131582cbfeb761d1f7e839717b9136 Oct 11 10:29:22.314437 master-1 kubenswrapper[4771]: I1011 10:29:22.314167 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-marketplace-xkrc6"] Oct 11 10:29:22.315607 master-1 kubenswrapper[4771]: I1011 10:29:22.315568 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.321530 master-1 kubenswrapper[4771]: I1011 10:29:22.321465 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-xkrc6"] Oct 11 10:29:22.335654 master-1 kubenswrapper[4771]: I1011 10:29:22.335578 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-gwwz9" event={"ID":"0b7d1d62-0062-47cd-a963-63893777198e","Type":"ContainerStarted","Data":"94df55f9d42e35f3eb12d9d840811113835d067c33b17a8f7670d61e212cd7f3"} Oct 11 10:29:22.338010 master-1 kubenswrapper[4771]: I1011 10:29:22.337960 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fgjvw" event={"ID":"2c084572-a5c9-4787-8a14-b7d6b0810a1b","Type":"ContainerStarted","Data":"a3c2aac5746fe934f86d97bbbc78969442131582cbfeb761d1f7e839717b9136"} Oct 11 10:29:22.385457 master-1 kubenswrapper[4771]: I1011 10:29:22.385376 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-oauth-apiserver_apiserver-65b6f4d4c9-skwvw_004ee387-d0e9-4582-ad14-f571832ebd6e/oauth-apiserver/0.log" Oct 11 10:29:22.436672 master-1 kubenswrapper[4771]: I1011 10:29:22.436587 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqnhk\" (UniqueName: \"kubernetes.io/projected/26005893-ecd8-4acb-8417-71a97ed97cbe-kube-api-access-hqnhk\") pod \"redhat-marketplace-xkrc6\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.436960 master-1 kubenswrapper[4771]: I1011 10:29:22.436721 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-catalog-content\") pod \"redhat-marketplace-xkrc6\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.436960 master-1 kubenswrapper[4771]: I1011 10:29:22.436739 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-utilities\") pod \"redhat-marketplace-xkrc6\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.537494 master-1 kubenswrapper[4771]: I1011 10:29:22.537398 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-catalog-content\") pod \"redhat-marketplace-xkrc6\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.537494 master-1 kubenswrapper[4771]: I1011 10:29:22.537479 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-utilities\") pod \"redhat-marketplace-xkrc6\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.537826 master-1 kubenswrapper[4771]: I1011 10:29:22.537597 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hqnhk\" (UniqueName: \"kubernetes.io/projected/26005893-ecd8-4acb-8417-71a97ed97cbe-kube-api-access-hqnhk\") pod \"redhat-marketplace-xkrc6\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.538563 master-1 kubenswrapper[4771]: I1011 10:29:22.538498 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-catalog-content\") pod \"redhat-marketplace-xkrc6\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.539094 master-1 kubenswrapper[4771]: I1011 10:29:22.538996 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-utilities\") pod \"redhat-marketplace-xkrc6\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.569665 master-1 kubenswrapper[4771]: I1011 10:29:22.569620 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqnhk\" (UniqueName: \"kubernetes.io/projected/26005893-ecd8-4acb-8417-71a97ed97cbe-kube-api-access-hqnhk\") pod \"redhat-marketplace-xkrc6\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.640411 master-1 kubenswrapper[4771]: I1011 10:29:22.640291 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:22.918491 master-1 kubenswrapper[4771]: I1011 10:29:22.918296 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm"] Oct 11 10:29:22.919105 master-1 kubenswrapper[4771]: I1011 10:29:22.919065 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:22.922843 master-1 kubenswrapper[4771]: I1011 10:29:22.922783 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-operator-lifecycle-manager"/"packageserver-service-cert" Oct 11 10:29:22.922910 master-1 kubenswrapper[4771]: I1011 10:29:22.922888 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-operator-lifecycle-manager"/"kube-root-ca.crt" Oct 11 10:29:22.923037 master-1 kubenswrapper[4771]: I1011 10:29:22.922787 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-operator-lifecycle-manager"/"openshift-service-ca.crt" Oct 11 10:29:22.929275 master-1 kubenswrapper[4771]: I1011 10:29:22.929235 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm"] Oct 11 10:29:23.043169 master-1 kubenswrapper[4771]: I1011 10:29:23.043051 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmpfs\" (UniqueName: \"kubernetes.io/empty-dir/68bdaf37-fa14-4c86-a697-881df7c9c7f1-tmpfs\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.043169 master-1 kubenswrapper[4771]: I1011 10:29:23.043179 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/68bdaf37-fa14-4c86-a697-881df7c9c7f1-apiservice-cert\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.043694 master-1 kubenswrapper[4771]: I1011 10:29:23.043284 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhgl\" (UniqueName: \"kubernetes.io/projected/68bdaf37-fa14-4c86-a697-881df7c9c7f1-kube-api-access-sdhgl\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.043694 master-1 kubenswrapper[4771]: I1011 10:29:23.043467 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68bdaf37-fa14-4c86-a697-881df7c9c7f1-webhook-cert\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.089979 master-1 kubenswrapper[4771]: I1011 10:29:23.089898 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-xkrc6"] Oct 11 10:29:23.101195 master-1 kubenswrapper[4771]: W1011 10:29:23.101115 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26005893_ecd8_4acb_8417_71a97ed97cbe.slice/crio-42678277150d23882615afd583505d1ee80fbc936870ab20c76affe3a676bd4c WatchSource:0}: Error finding container 42678277150d23882615afd583505d1ee80fbc936870ab20c76affe3a676bd4c: Status 404 returned error can't find the container with id 42678277150d23882615afd583505d1ee80fbc936870ab20c76affe3a676bd4c Oct 11 10:29:23.144379 master-1 kubenswrapper[4771]: I1011 10:29:23.144293 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/68bdaf37-fa14-4c86-a697-881df7c9c7f1-apiservice-cert\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.144379 master-1 kubenswrapper[4771]: I1011 10:29:23.144370 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhgl\" (UniqueName: \"kubernetes.io/projected/68bdaf37-fa14-4c86-a697-881df7c9c7f1-kube-api-access-sdhgl\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.144699 master-1 kubenswrapper[4771]: I1011 10:29:23.144431 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68bdaf37-fa14-4c86-a697-881df7c9c7f1-webhook-cert\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.144699 master-1 kubenswrapper[4771]: I1011 10:29:23.144516 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"tmpfs\" (UniqueName: \"kubernetes.io/empty-dir/68bdaf37-fa14-4c86-a697-881df7c9c7f1-tmpfs\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.145305 master-1 kubenswrapper[4771]: I1011 10:29:23.145264 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"tmpfs\" (UniqueName: \"kubernetes.io/empty-dir/68bdaf37-fa14-4c86-a697-881df7c9c7f1-tmpfs\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.148307 master-1 kubenswrapper[4771]: I1011 10:29:23.148262 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/68bdaf37-fa14-4c86-a697-881df7c9c7f1-apiservice-cert\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.148902 master-1 kubenswrapper[4771]: I1011 10:29:23.148859 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68bdaf37-fa14-4c86-a697-881df7c9c7f1-webhook-cert\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.165713 master-1 kubenswrapper[4771]: I1011 10:29:23.165177 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhgl\" (UniqueName: \"kubernetes.io/projected/68bdaf37-fa14-4c86-a697-881df7c9c7f1-kube-api-access-sdhgl\") pod \"packageserver-77c85f5c6-6zxmm\" (UID: \"68bdaf37-fa14-4c86-a697-881df7c9c7f1\") " pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.234304 master-1 kubenswrapper[4771]: I1011 10:29:23.234164 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:23.345839 master-1 kubenswrapper[4771]: I1011 10:29:23.345039 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-xkrc6" event={"ID":"26005893-ecd8-4acb-8417-71a97ed97cbe","Type":"ContainerStarted","Data":"42678277150d23882615afd583505d1ee80fbc936870ab20c76affe3a676bd4c"} Oct 11 10:29:23.520929 master-1 kubenswrapper[4771]: I1011 10:29:23.520762 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-operators-g8tm6"] Oct 11 10:29:23.521518 master-1 kubenswrapper[4771]: I1011 10:29:23.521483 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.531988 master-1 kubenswrapper[4771]: I1011 10:29:23.530927 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-g8tm6"] Oct 11 10:29:23.649335 master-1 kubenswrapper[4771]: I1011 10:29:23.649276 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-utilities\") pod \"redhat-operators-g8tm6\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.649623 master-1 kubenswrapper[4771]: I1011 10:29:23.649350 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpmjh\" (UniqueName: \"kubernetes.io/projected/38131fcf-d407-4ba3-b7bf-471586bab887-kube-api-access-gpmjh\") pod \"redhat-operators-g8tm6\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.649623 master-1 kubenswrapper[4771]: I1011 10:29:23.649448 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-catalog-content\") pod \"redhat-operators-g8tm6\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.650421 master-1 kubenswrapper[4771]: I1011 10:29:23.650374 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm"] Oct 11 10:29:23.657428 master-1 kubenswrapper[4771]: W1011 10:29:23.657339 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bdaf37_fa14_4c86_a697_881df7c9c7f1.slice/crio-4a2c543ad15bd096a76d71db493ea6b2ffca6a47d70401e74955a5901737d9c3 WatchSource:0}: Error finding container 4a2c543ad15bd096a76d71db493ea6b2ffca6a47d70401e74955a5901737d9c3: Status 404 returned error can't find the container with id 4a2c543ad15bd096a76d71db493ea6b2ffca6a47d70401e74955a5901737d9c3 Oct 11 10:29:23.750502 master-1 kubenswrapper[4771]: I1011 10:29:23.750438 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-utilities\") pod \"redhat-operators-g8tm6\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.750502 master-1 kubenswrapper[4771]: I1011 10:29:23.750501 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmjh\" (UniqueName: \"kubernetes.io/projected/38131fcf-d407-4ba3-b7bf-471586bab887-kube-api-access-gpmjh\") pod \"redhat-operators-g8tm6\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.750757 master-1 kubenswrapper[4771]: I1011 10:29:23.750529 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-catalog-content\") pod \"redhat-operators-g8tm6\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.751099 master-1 kubenswrapper[4771]: I1011 10:29:23.751071 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-catalog-content\") pod \"redhat-operators-g8tm6\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.751466 master-1 kubenswrapper[4771]: I1011 10:29:23.751412 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-utilities\") pod \"redhat-operators-g8tm6\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.773022 master-1 kubenswrapper[4771]: I1011 10:29:23.772923 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmjh\" (UniqueName: \"kubernetes.io/projected/38131fcf-d407-4ba3-b7bf-471586bab887-kube-api-access-gpmjh\") pod \"redhat-operators-g8tm6\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:23.839927 master-1 kubenswrapper[4771]: I1011 10:29:23.839855 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:24.153809 master-1 kubenswrapper[4771]: I1011 10:29:24.153746 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:29:24.154085 master-1 kubenswrapper[4771]: E1011 10:29:24.153945 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:29:24.154085 master-1 kubenswrapper[4771]: E1011 10:29:24.154082 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:29:56.154057698 +0000 UTC m=+228.128284149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:29:24.224098 master-1 kubenswrapper[4771]: I1011 10:29:24.224017 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-g8tm6"] Oct 11 10:29:24.265962 master-1 kubenswrapper[4771]: W1011 10:29:24.265887 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38131fcf_d407_4ba3_b7bf_471586bab887.slice/crio-fd83c4d331d341ca058f07884e0c753dec2509d54999da528657ce66ee47354c WatchSource:0}: Error finding container fd83c4d331d341ca058f07884e0c753dec2509d54999da528657ce66ee47354c: Status 404 returned error can't find the container with id fd83c4d331d341ca058f07884e0c753dec2509d54999da528657ce66ee47354c Oct 11 10:29:24.351323 master-1 kubenswrapper[4771]: I1011 10:29:24.351239 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" event={"ID":"68bdaf37-fa14-4c86-a697-881df7c9c7f1","Type":"ContainerStarted","Data":"4a2c543ad15bd096a76d71db493ea6b2ffca6a47d70401e74955a5901737d9c3"} Oct 11 10:29:24.353056 master-1 kubenswrapper[4771]: I1011 10:29:24.353007 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-g8tm6" event={"ID":"38131fcf-d407-4ba3-b7bf-471586bab887","Type":"ContainerStarted","Data":"fd83c4d331d341ca058f07884e0c753dec2509d54999da528657ce66ee47354c"} Oct 11 10:29:24.378689 master-1 kubenswrapper[4771]: I1011 10:29:24.378555 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzjcf_b3f49f37-a9e4-4acd-ae7e-d644e8475106/dns/0.log" Oct 11 10:29:24.579521 master-1 kubenswrapper[4771]: I1011 10:29:24.579470 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzjcf_b3f49f37-a9e4-4acd-ae7e-d644e8475106/kube-rbac-proxy/0.log" Oct 11 10:29:25.175179 master-1 kubenswrapper[4771]: I1011 10:29:25.174703 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5ddb89f76-z5t6x"] Oct 11 10:29:25.177590 master-1 kubenswrapper[4771]: I1011 10:29:25.175578 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.177590 master-1 kubenswrapper[4771]: I1011 10:29:25.175758 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps"] Oct 11 10:29:25.177590 master-1 kubenswrapper[4771]: I1011 10:29:25.176331 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" Oct 11 10:29:25.178427 master-1 kubenswrapper[4771]: I1011 10:29:25.178380 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-ingress"/"router-metrics-certs-default" Oct 11 10:29:25.179427 master-1 kubenswrapper[4771]: I1011 10:29:25.179349 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fjwjw_2919a957-a46f-4e96-b42e-3ba3c537e98e/dns-node-resolver/0.log" Oct 11 10:29:25.179534 master-1 kubenswrapper[4771]: I1011 10:29:25.179496 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ingress"/"openshift-service-ca.crt" Oct 11 10:29:25.179582 master-1 kubenswrapper[4771]: I1011 10:29:25.179494 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-ingress"/"router-certs-default" Oct 11 10:29:25.179582 master-1 kubenswrapper[4771]: I1011 10:29:25.179548 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ingress"/"kube-root-ca.crt" Oct 11 10:29:25.183473 master-1 kubenswrapper[4771]: I1011 10:29:25.179680 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ingress"/"service-ca-bundle" Oct 11 10:29:25.183473 master-1 kubenswrapper[4771]: I1011 10:29:25.179719 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-ingress"/"router-stats-default" Oct 11 10:29:25.183473 master-1 kubenswrapper[4771]: I1011 10:29:25.179775 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"prometheus-operator-admission-webhook-tls" Oct 11 10:29:25.183473 master-1 kubenswrapper[4771]: I1011 10:29:25.183261 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-967c7bb47-djx82"] Oct 11 10:29:25.189015 master-1 kubenswrapper[4771]: I1011 10:29:25.184110 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-967c7bb47-djx82" Oct 11 10:29:25.189015 master-1 kubenswrapper[4771]: I1011 10:29:25.186320 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps"] Oct 11 10:29:25.189270 master-1 kubenswrapper[4771]: I1011 10:29:25.189115 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-967c7bb47-djx82"] Oct 11 10:29:25.270155 master-1 kubenswrapper[4771]: I1011 10:29:25.270069 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cd4a19-2532-43d1-9144-1f59d9e52d19-metrics-certs\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.270349 master-1 kubenswrapper[4771]: I1011 10:29:25.270199 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xh9q\" (UniqueName: \"kubernetes.io/projected/abd3363b-056a-4468-b13b-3e353929307d-kube-api-access-5xh9q\") pod \"network-check-source-967c7bb47-djx82\" (UID: \"abd3363b-056a-4468-b13b-3e353929307d\") " pod="openshift-network-diagnostics/network-check-source-967c7bb47-djx82" Oct 11 10:29:25.270349 master-1 kubenswrapper[4771]: I1011 10:29:25.270244 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2631acfc-dace-435d-8ea9-65d023c13ab6-tls-certificates\") pod \"prometheus-operator-admission-webhook-79d5f95f5c-67qps\" (UID: \"2631acfc-dace-435d-8ea9-65d023c13ab6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" Oct 11 10:29:25.270349 master-1 kubenswrapper[4771]: I1011 10:29:25.270284 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/04cd4a19-2532-43d1-9144-1f59d9e52d19-default-certificate\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.270490 master-1 kubenswrapper[4771]: I1011 10:29:25.270385 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04cd4a19-2532-43d1-9144-1f59d9e52d19-service-ca-bundle\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.270521 master-1 kubenswrapper[4771]: I1011 10:29:25.270484 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdbm\" (UniqueName: \"kubernetes.io/projected/04cd4a19-2532-43d1-9144-1f59d9e52d19-kube-api-access-7gdbm\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.270548 master-1 kubenswrapper[4771]: I1011 10:29:25.270523 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/04cd4a19-2532-43d1-9144-1f59d9e52d19-stats-auth\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.360413 master-1 kubenswrapper[4771]: I1011 10:29:25.360283 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fgjvw" event={"ID":"2c084572-a5c9-4787-8a14-b7d6b0810a1b","Type":"ContainerStarted","Data":"fccb120465597209818f91c776fbabe2d81f28a21944709ed07033cb4785774c"} Oct 11 10:29:25.360413 master-1 kubenswrapper[4771]: I1011 10:29:25.360425 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fgjvw" event={"ID":"2c084572-a5c9-4787-8a14-b7d6b0810a1b","Type":"ContainerStarted","Data":"4ad7978bcb64b587f8b7811ba06ddea362e2663fe0567625b02089ee562de4a3"} Oct 11 10:29:25.372139 master-1 kubenswrapper[4771]: I1011 10:29:25.372048 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/04cd4a19-2532-43d1-9144-1f59d9e52d19-stats-auth\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.372331 master-1 kubenswrapper[4771]: I1011 10:29:25.372179 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cd4a19-2532-43d1-9144-1f59d9e52d19-metrics-certs\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.372331 master-1 kubenswrapper[4771]: I1011 10:29:25.372297 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5xh9q\" (UniqueName: \"kubernetes.io/projected/abd3363b-056a-4468-b13b-3e353929307d-kube-api-access-5xh9q\") pod \"network-check-source-967c7bb47-djx82\" (UID: \"abd3363b-056a-4468-b13b-3e353929307d\") " pod="openshift-network-diagnostics/network-check-source-967c7bb47-djx82" Oct 11 10:29:25.372476 master-1 kubenswrapper[4771]: I1011 10:29:25.372348 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2631acfc-dace-435d-8ea9-65d023c13ab6-tls-certificates\") pod \"prometheus-operator-admission-webhook-79d5f95f5c-67qps\" (UID: \"2631acfc-dace-435d-8ea9-65d023c13ab6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" Oct 11 10:29:25.372528 master-1 kubenswrapper[4771]: I1011 10:29:25.372503 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/04cd4a19-2532-43d1-9144-1f59d9e52d19-default-certificate\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.372643 master-1 kubenswrapper[4771]: I1011 10:29:25.372597 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04cd4a19-2532-43d1-9144-1f59d9e52d19-service-ca-bundle\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.372742 master-1 kubenswrapper[4771]: I1011 10:29:25.372705 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdbm\" (UniqueName: \"kubernetes.io/projected/04cd4a19-2532-43d1-9144-1f59d9e52d19-kube-api-access-7gdbm\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.373776 master-1 kubenswrapper[4771]: I1011 10:29:25.373734 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04cd4a19-2532-43d1-9144-1f59d9e52d19-service-ca-bundle\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.376750 master-1 kubenswrapper[4771]: I1011 10:29:25.376708 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/04cd4a19-2532-43d1-9144-1f59d9e52d19-default-certificate\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.377155 master-1 kubenswrapper[4771]: I1011 10:29:25.377111 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2631acfc-dace-435d-8ea9-65d023c13ab6-tls-certificates\") pod \"prometheus-operator-admission-webhook-79d5f95f5c-67qps\" (UID: \"2631acfc-dace-435d-8ea9-65d023c13ab6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" Oct 11 10:29:25.378023 master-1 kubenswrapper[4771]: I1011 10:29:25.377974 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fgjvw" podStartSLOduration=130.370612377 podStartE2EDuration="2m12.377962316s" podCreationTimestamp="2025-10-11 10:27:13 +0000 UTC" firstStartedPulling="2025-10-11 10:29:22.293522286 +0000 UTC m=+194.267748727" lastFinishedPulling="2025-10-11 10:29:24.300872205 +0000 UTC m=+196.275098666" observedRunningTime="2025-10-11 10:29:25.376387962 +0000 UTC m=+197.350614483" watchObservedRunningTime="2025-10-11 10:29:25.377962316 +0000 UTC m=+197.352188757" Oct 11 10:29:25.379347 master-1 kubenswrapper[4771]: I1011 10:29:25.379308 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/04cd4a19-2532-43d1-9144-1f59d9e52d19-stats-auth\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.383410 master-1 kubenswrapper[4771]: I1011 10:29:25.383369 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cd4a19-2532-43d1-9144-1f59d9e52d19-metrics-certs\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.403479 master-1 kubenswrapper[4771]: I1011 10:29:25.403410 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xh9q\" (UniqueName: \"kubernetes.io/projected/abd3363b-056a-4468-b13b-3e353929307d-kube-api-access-5xh9q\") pod \"network-check-source-967c7bb47-djx82\" (UID: \"abd3363b-056a-4468-b13b-3e353929307d\") " pod="openshift-network-diagnostics/network-check-source-967c7bb47-djx82" Oct 11 10:29:25.407416 master-1 kubenswrapper[4771]: I1011 10:29:25.407334 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdbm\" (UniqueName: \"kubernetes.io/projected/04cd4a19-2532-43d1-9144-1f59d9e52d19-kube-api-access-7gdbm\") pod \"router-default-5ddb89f76-z5t6x\" (UID: \"04cd4a19-2532-43d1-9144-1f59d9e52d19\") " pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.493967 master-1 kubenswrapper[4771]: I1011 10:29:25.493771 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:25.506480 master-1 kubenswrapper[4771]: I1011 10:29:25.506411 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" Oct 11 10:29:25.513386 master-1 kubenswrapper[4771]: I1011 10:29:25.513300 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-967c7bb47-djx82" Oct 11 10:29:25.541096 master-1 kubenswrapper[4771]: W1011 10:29:25.541034 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cd4a19_2532_43d1_9144_1f59d9e52d19.slice/crio-ffb197fb6a875de889d24d5f079ca17b067a86a2db866b6b6480cc1cd47ddea6 WatchSource:0}: Error finding container ffb197fb6a875de889d24d5f079ca17b067a86a2db866b6b6480cc1cd47ddea6: Status 404 returned error can't find the container with id ffb197fb6a875de889d24d5f079ca17b067a86a2db866b6b6480cc1cd47ddea6 Oct 11 10:29:25.780033 master-1 kubenswrapper[4771]: I1011 10:29:25.779908 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_installer-1-master-1_826e1279-bc0d-426e-b6e0-5108268f340e/installer/0.log" Oct 11 10:29:25.900290 master-1 kubenswrapper[4771]: I1011 10:29:25.900238 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps"] Oct 11 10:29:25.906644 master-1 kubenswrapper[4771]: W1011 10:29:25.906597 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2631acfc_dace_435d_8ea9_65d023c13ab6.slice/crio-7903dcdfa0035252353a54ffe91b977075a9669460965531b40fe22deec61a78 WatchSource:0}: Error finding container 7903dcdfa0035252353a54ffe91b977075a9669460965531b40fe22deec61a78: Status 404 returned error can't find the container with id 7903dcdfa0035252353a54ffe91b977075a9669460965531b40fe22deec61a78 Oct 11 10:29:25.962607 master-1 kubenswrapper[4771]: I1011 10:29:25.962544 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-967c7bb47-djx82"] Oct 11 10:29:25.968952 master-1 kubenswrapper[4771]: W1011 10:29:25.968906 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd3363b_056a_4468_b13b_3e353929307d.slice/crio-ca29caeaf1e6bc39a022ff46244647d6149803f865e5c9c560c81008d170c904 WatchSource:0}: Error finding container ca29caeaf1e6bc39a022ff46244647d6149803f865e5c9c560c81008d170c904: Status 404 returned error can't find the container with id ca29caeaf1e6bc39a022ff46244647d6149803f865e5c9c560c81008d170c904 Oct 11 10:29:26.372533 master-1 kubenswrapper[4771]: I1011 10:29:26.372472 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-967c7bb47-djx82" event={"ID":"abd3363b-056a-4468-b13b-3e353929307d","Type":"ContainerStarted","Data":"dd0f391c28ea252aea31150e5c7ca0630e5b84d86fe5527ce669a76f5c8eb413"} Oct 11 10:29:26.373117 master-1 kubenswrapper[4771]: I1011 10:29:26.372544 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-967c7bb47-djx82" event={"ID":"abd3363b-056a-4468-b13b-3e353929307d","Type":"ContainerStarted","Data":"ca29caeaf1e6bc39a022ff46244647d6149803f865e5c9c560c81008d170c904"} Oct 11 10:29:26.374406 master-1 kubenswrapper[4771]: I1011 10:29:26.374377 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" event={"ID":"2631acfc-dace-435d-8ea9-65d023c13ab6","Type":"ContainerStarted","Data":"7903dcdfa0035252353a54ffe91b977075a9669460965531b40fe22deec61a78"} Oct 11 10:29:26.375878 master-1 kubenswrapper[4771]: I1011 10:29:26.375807 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" event={"ID":"04cd4a19-2532-43d1-9144-1f59d9e52d19","Type":"ContainerStarted","Data":"ffb197fb6a875de889d24d5f079ca17b067a86a2db866b6b6480cc1cd47ddea6"} Oct 11 10:29:26.397130 master-1 kubenswrapper[4771]: I1011 10:29:26.397052 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-967c7bb47-djx82" podStartSLOduration=119.397033138 podStartE2EDuration="1m59.397033138s" podCreationTimestamp="2025-10-11 10:27:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:26.395812574 +0000 UTC m=+198.370039035" watchObservedRunningTime="2025-10-11 10:29:26.397033138 +0000 UTC m=+198.371259579" Oct 11 10:29:26.583459 master-1 kubenswrapper[4771]: I1011 10:29:26.583378 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:29:26.583739 master-1 kubenswrapper[4771]: E1011 10:29:26.583474 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:29:26.583739 master-1 kubenswrapper[4771]: E1011 10:29:26.583589 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:29:58.58356456 +0000 UTC m=+230.557791041 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:29:26.979870 master-1 kubenswrapper[4771]: I1011 10:29:26.979819 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-1-master-1_6534d9db-a553-4c39-bf4a-014a359ee336/installer/0.log" Oct 11 10:29:27.381382 master-1 kubenswrapper[4771]: I1011 10:29:27.381313 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_installer-4-master-1_7662f87a-13ba-439c-b386-05e68284803c/installer/0.log" Oct 11 10:29:27.934645 master-1 kubenswrapper[4771]: I1011 10:29:27.934560 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:29:27.936646 master-1 kubenswrapper[4771]: I1011 10:29:27.936373 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:29:27.973581 master-1 kubenswrapper[4771]: I1011 10:29:27.973547 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:29:28.098220 master-1 kubenswrapper[4771]: I1011 10:29:28.098098 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-usr-local-bin\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.098220 master-1 kubenswrapper[4771]: I1011 10:29:28.098163 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-cert-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.098220 master-1 kubenswrapper[4771]: I1011 10:29:28.098184 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-data-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.098721 master-1 kubenswrapper[4771]: I1011 10:29:28.098259 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-resource-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.098721 master-1 kubenswrapper[4771]: I1011 10:29:28.098287 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-log-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.098721 master-1 kubenswrapper[4771]: I1011 10:29:28.098345 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-static-pod-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199346 master-1 kubenswrapper[4771]: I1011 10:29:28.199157 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-resource-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199346 master-1 kubenswrapper[4771]: I1011 10:29:28.199255 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-log-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199346 master-1 kubenswrapper[4771]: I1011 10:29:28.199286 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-resource-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199346 master-1 kubenswrapper[4771]: I1011 10:29:28.199336 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-static-pod-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199661 master-1 kubenswrapper[4771]: I1011 10:29:28.199384 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-static-pod-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199661 master-1 kubenswrapper[4771]: I1011 10:29:28.199415 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-log-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199661 master-1 kubenswrapper[4771]: I1011 10:29:28.199449 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-usr-local-bin\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199661 master-1 kubenswrapper[4771]: I1011 10:29:28.199517 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-usr-local-bin\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199661 master-1 kubenswrapper[4771]: I1011 10:29:28.199594 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-data-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199661 master-1 kubenswrapper[4771]: I1011 10:29:28.199644 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-cert-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199823 master-1 kubenswrapper[4771]: I1011 10:29:28.199679 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-data-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.199823 master-1 kubenswrapper[4771]: I1011 10:29:28.199746 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-cert-dir\") pod \"etcd-master-1\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.271754 master-1 kubenswrapper[4771]: I1011 10:29:28.271658 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:29:28.384749 master-1 kubenswrapper[4771]: I1011 10:29:28.384693 4771 generic.go:334] "Generic (PLEG): container finished" podID="826e1279-bc0d-426e-b6e0-5108268f340e" containerID="9a616ae6ac6ffcbc27ae54a54aec1c65046926d3773ee73ab8bfdedb75371f06" exitCode=0 Oct 11 10:29:28.385406 master-1 kubenswrapper[4771]: I1011 10:29:28.384749 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-1-master-1" event={"ID":"826e1279-bc0d-426e-b6e0-5108268f340e","Type":"ContainerDied","Data":"9a616ae6ac6ffcbc27ae54a54aec1c65046926d3773ee73ab8bfdedb75371f06"} Oct 11 10:29:29.529431 master-1 kubenswrapper[4771]: I1011 10:29:29.526779 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/machine-config-server-h7gnk"] Oct 11 10:29:29.529431 master-1 kubenswrapper[4771]: I1011 10:29:29.527324 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.532250 master-1 kubenswrapper[4771]: I1011 10:29:29.532169 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-machine-config-operator"/"machine-config-server-tls" Oct 11 10:29:29.532729 master-1 kubenswrapper[4771]: I1011 10:29:29.532679 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-machine-config-operator"/"node-bootstrapper-token" Oct 11 10:29:29.716347 master-1 kubenswrapper[4771]: I1011 10:29:29.716176 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgpg7\" (UniqueName: \"kubernetes.io/projected/6d20faa4-e5eb-4766-b4f5-30e491d1820c-kube-api-access-sgpg7\") pod \"machine-config-server-h7gnk\" (UID: \"6d20faa4-e5eb-4766-b4f5-30e491d1820c\") " pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.716347 master-1 kubenswrapper[4771]: I1011 10:29:29.716290 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-bootstrap-token\" (UniqueName: \"kubernetes.io/secret/6d20faa4-e5eb-4766-b4f5-30e491d1820c-node-bootstrap-token\") pod \"machine-config-server-h7gnk\" (UID: \"6d20faa4-e5eb-4766-b4f5-30e491d1820c\") " pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.716622 master-1 kubenswrapper[4771]: I1011 10:29:29.716400 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certs\" (UniqueName: \"kubernetes.io/secret/6d20faa4-e5eb-4766-b4f5-30e491d1820c-certs\") pod \"machine-config-server-h7gnk\" (UID: \"6d20faa4-e5eb-4766-b4f5-30e491d1820c\") " pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.817570 master-1 kubenswrapper[4771]: I1011 10:29:29.817470 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"certs\" (UniqueName: \"kubernetes.io/secret/6d20faa4-e5eb-4766-b4f5-30e491d1820c-certs\") pod \"machine-config-server-h7gnk\" (UID: \"6d20faa4-e5eb-4766-b4f5-30e491d1820c\") " pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.817695 master-1 kubenswrapper[4771]: I1011 10:29:29.817570 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-sgpg7\" (UniqueName: \"kubernetes.io/projected/6d20faa4-e5eb-4766-b4f5-30e491d1820c-kube-api-access-sgpg7\") pod \"machine-config-server-h7gnk\" (UID: \"6d20faa4-e5eb-4766-b4f5-30e491d1820c\") " pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.817695 master-1 kubenswrapper[4771]: I1011 10:29:29.817644 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-bootstrap-token\" (UniqueName: \"kubernetes.io/secret/6d20faa4-e5eb-4766-b4f5-30e491d1820c-node-bootstrap-token\") pod \"machine-config-server-h7gnk\" (UID: \"6d20faa4-e5eb-4766-b4f5-30e491d1820c\") " pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.822445 master-1 kubenswrapper[4771]: I1011 10:29:29.822381 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-bootstrap-token\" (UniqueName: \"kubernetes.io/secret/6d20faa4-e5eb-4766-b4f5-30e491d1820c-node-bootstrap-token\") pod \"machine-config-server-h7gnk\" (UID: \"6d20faa4-e5eb-4766-b4f5-30e491d1820c\") " pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.822579 master-1 kubenswrapper[4771]: I1011 10:29:29.822463 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"certs\" (UniqueName: \"kubernetes.io/secret/6d20faa4-e5eb-4766-b4f5-30e491d1820c-certs\") pod \"machine-config-server-h7gnk\" (UID: \"6d20faa4-e5eb-4766-b4f5-30e491d1820c\") " pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.834844 master-1 kubenswrapper[4771]: I1011 10:29:29.834757 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgpg7\" (UniqueName: \"kubernetes.io/projected/6d20faa4-e5eb-4766-b4f5-30e491d1820c-kube-api-access-sgpg7\") pod \"machine-config-server-h7gnk\" (UID: \"6d20faa4-e5eb-4766-b4f5-30e491d1820c\") " pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.846658 master-1 kubenswrapper[4771]: I1011 10:29:29.846585 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/machine-config-server-h7gnk" Oct 11 10:29:29.943990 master-1 kubenswrapper[4771]: W1011 10:29:29.943916 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5268b2f2ae2aef0c7f2e7a6e651ed702.slice/crio-da72a8df71c43223b153f8d9058eb065866400eec698b95e777f1a50e9811194 WatchSource:0}: Error finding container da72a8df71c43223b153f8d9058eb065866400eec698b95e777f1a50e9811194: Status 404 returned error can't find the container with id da72a8df71c43223b153f8d9058eb065866400eec698b95e777f1a50e9811194 Oct 11 10:29:29.951115 master-1 kubenswrapper[4771]: W1011 10:29:29.951056 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d20faa4_e5eb_4766_b4f5_30e491d1820c.slice/crio-d9139183c94d28fcf1a4d7ca8fca07b3415c016dbb4dcf57c4b77203784c50aa WatchSource:0}: Error finding container d9139183c94d28fcf1a4d7ca8fca07b3415c016dbb4dcf57c4b77203784c50aa: Status 404 returned error can't find the container with id d9139183c94d28fcf1a4d7ca8fca07b3415c016dbb4dcf57c4b77203784c50aa Oct 11 10:29:29.969079 master-1 kubenswrapper[4771]: I1011 10:29:29.968963 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-1-master-1" Oct 11 10:29:29.978286 master-1 kubenswrapper[4771]: I1011 10:29:29.978225 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-apiserver_apiserver-555f658fd6-n5n6g_027736d1-f3d3-490e-9ee1-d08bad7a25b7/fix-audit-permissions/0.log" Oct 11 10:29:30.121265 master-1 kubenswrapper[4771]: I1011 10:29:30.121115 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/826e1279-bc0d-426e-b6e0-5108268f340e-kube-api-access\") pod \"826e1279-bc0d-426e-b6e0-5108268f340e\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " Oct 11 10:29:30.121265 master-1 kubenswrapper[4771]: I1011 10:29:30.121192 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-var-lock\") pod \"826e1279-bc0d-426e-b6e0-5108268f340e\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " Oct 11 10:29:30.121265 master-1 kubenswrapper[4771]: I1011 10:29:30.121230 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-kubelet-dir\") pod \"826e1279-bc0d-426e-b6e0-5108268f340e\" (UID: \"826e1279-bc0d-426e-b6e0-5108268f340e\") " Oct 11 10:29:30.121588 master-1 kubenswrapper[4771]: I1011 10:29:30.121368 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-var-lock" (OuterVolumeSpecName: "var-lock") pod "826e1279-bc0d-426e-b6e0-5108268f340e" (UID: "826e1279-bc0d-426e-b6e0-5108268f340e"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:30.121588 master-1 kubenswrapper[4771]: I1011 10:29:30.121453 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "826e1279-bc0d-426e-b6e0-5108268f340e" (UID: "826e1279-bc0d-426e-b6e0-5108268f340e"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:30.126016 master-1 kubenswrapper[4771]: I1011 10:29:30.125941 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826e1279-bc0d-426e-b6e0-5108268f340e-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "826e1279-bc0d-426e-b6e0-5108268f340e" (UID: "826e1279-bc0d-426e-b6e0-5108268f340e"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:29:30.183791 master-1 kubenswrapper[4771]: I1011 10:29:30.183665 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-apiserver_apiserver-555f658fd6-n5n6g_027736d1-f3d3-490e-9ee1-d08bad7a25b7/openshift-apiserver/0.log" Oct 11 10:29:30.223183 master-1 kubenswrapper[4771]: I1011 10:29:30.222970 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/826e1279-bc0d-426e-b6e0-5108268f340e-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:30.223183 master-1 kubenswrapper[4771]: I1011 10:29:30.223018 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:30.223183 master-1 kubenswrapper[4771]: I1011 10:29:30.223032 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/826e1279-bc0d-426e-b6e0-5108268f340e-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:30.381064 master-1 kubenswrapper[4771]: I1011 10:29:30.381006 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-apiserver_apiserver-555f658fd6-n5n6g_027736d1-f3d3-490e-9ee1-d08bad7a25b7/openshift-apiserver-check-endpoints/0.log" Oct 11 10:29:30.393240 master-1 kubenswrapper[4771]: I1011 10:29:30.393204 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-1-master-1" Oct 11 10:29:30.393423 master-1 kubenswrapper[4771]: I1011 10:29:30.393188 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-1-master-1" event={"ID":"826e1279-bc0d-426e-b6e0-5108268f340e","Type":"ContainerDied","Data":"5363fbd5b12a9230e5f6b1dd57b8fb070fa12eb536a76d7bdfd11f7b2167cad9"} Oct 11 10:29:30.393488 master-1 kubenswrapper[4771]: I1011 10:29:30.393436 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5363fbd5b12a9230e5f6b1dd57b8fb070fa12eb536a76d7bdfd11f7b2167cad9" Oct 11 10:29:30.395586 master-1 kubenswrapper[4771]: I1011 10:29:30.395547 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerStarted","Data":"da72a8df71c43223b153f8d9058eb065866400eec698b95e777f1a50e9811194"} Oct 11 10:29:30.397334 master-1 kubenswrapper[4771]: I1011 10:29:30.397291 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/machine-config-server-h7gnk" event={"ID":"6d20faa4-e5eb-4766-b4f5-30e491d1820c","Type":"ContainerStarted","Data":"faf96c0ce971191959bb91c959b4ff7b528d62bb5ffc85249401a032257b3608"} Oct 11 10:29:30.397425 master-1 kubenswrapper[4771]: I1011 10:29:30.397373 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/machine-config-server-h7gnk" event={"ID":"6d20faa4-e5eb-4766-b4f5-30e491d1820c","Type":"ContainerStarted","Data":"d9139183c94d28fcf1a4d7ca8fca07b3415c016dbb4dcf57c4b77203784c50aa"} Oct 11 10:29:30.399053 master-1 kubenswrapper[4771]: I1011 10:29:30.399010 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b7d1d62-0062-47cd-a963-63893777198e" containerID="87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865" exitCode=0 Oct 11 10:29:30.399127 master-1 kubenswrapper[4771]: I1011 10:29:30.399082 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-gwwz9" event={"ID":"0b7d1d62-0062-47cd-a963-63893777198e","Type":"ContainerDied","Data":"87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865"} Oct 11 10:29:30.400833 master-1 kubenswrapper[4771]: I1011 10:29:30.400587 4771 generic.go:334] "Generic (PLEG): container finished" podID="38131fcf-d407-4ba3-b7bf-471586bab887" containerID="46478dfa370c61d5e583543ca4a34b66afd1e95ecf434515eb16283cfe8a52de" exitCode=0 Oct 11 10:29:30.400833 master-1 kubenswrapper[4771]: I1011 10:29:30.400611 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-g8tm6" event={"ID":"38131fcf-d407-4ba3-b7bf-471586bab887","Type":"ContainerDied","Data":"46478dfa370c61d5e583543ca4a34b66afd1e95ecf434515eb16283cfe8a52de"} Oct 11 10:29:30.414396 master-1 kubenswrapper[4771]: I1011 10:29:30.414300 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/machine-config-server-h7gnk" podStartSLOduration=1.41427744 podStartE2EDuration="1.41427744s" podCreationTimestamp="2025-10-11 10:29:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:30.413294153 +0000 UTC m=+202.387520594" watchObservedRunningTime="2025-10-11 10:29:30.41427744 +0000 UTC m=+202.388503891" Oct 11 10:29:31.406198 master-1 kubenswrapper[4771]: I1011 10:29:31.406139 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" event={"ID":"68bdaf37-fa14-4c86-a697-881df7c9c7f1","Type":"ContainerStarted","Data":"a4abbedd79e929c2bd4cf2fc94f8f962fc6824e374c69e8840181904d932e12c"} Oct 11 10:29:31.406792 master-1 kubenswrapper[4771]: I1011 10:29:31.406416 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:31.408934 master-1 kubenswrapper[4771]: I1011 10:29:31.408850 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" event={"ID":"04cd4a19-2532-43d1-9144-1f59d9e52d19","Type":"ContainerStarted","Data":"d9d09acfb9b74efc71914e418c9f7ad84873a3a13515d6cfcddf159cfd555604"} Oct 11 10:29:31.410794 master-1 kubenswrapper[4771]: I1011 10:29:31.410775 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" Oct 11 10:29:31.412384 master-1 kubenswrapper[4771]: I1011 10:29:31.412343 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" event={"ID":"2631acfc-dace-435d-8ea9-65d023c13ab6","Type":"ContainerStarted","Data":"cdd4374bd67def2eb7eddbd9f2e5b7b6ce62ae68a06fcef83eba60f54d368831"} Oct 11 10:29:31.412612 master-1 kubenswrapper[4771]: I1011 10:29:31.412576 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" Oct 11 10:29:31.414166 master-1 kubenswrapper[4771]: I1011 10:29:31.414103 4771 generic.go:334] "Generic (PLEG): container finished" podID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerID="142add763393fde94b8ed6a34c3ef572a32e34909b409ad71cf3570c801fa30d" exitCode=0 Oct 11 10:29:31.414234 master-1 kubenswrapper[4771]: I1011 10:29:31.414176 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-xkrc6" event={"ID":"26005893-ecd8-4acb-8417-71a97ed97cbe","Type":"ContainerDied","Data":"142add763393fde94b8ed6a34c3ef572a32e34909b409ad71cf3570c801fa30d"} Oct 11 10:29:31.416867 master-1 kubenswrapper[4771]: I1011 10:29:31.416817 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" Oct 11 10:29:31.425937 master-1 kubenswrapper[4771]: I1011 10:29:31.425853 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operator-lifecycle-manager/packageserver-77c85f5c6-6zxmm" podStartSLOduration=2.7429406309999997 podStartE2EDuration="9.425831845s" podCreationTimestamp="2025-10-11 10:29:22 +0000 UTC" firstStartedPulling="2025-10-11 10:29:23.659928926 +0000 UTC m=+195.634155367" lastFinishedPulling="2025-10-11 10:29:30.34282014 +0000 UTC m=+202.317046581" observedRunningTime="2025-10-11 10:29:31.423623764 +0000 UTC m=+203.397850215" watchObservedRunningTime="2025-10-11 10:29:31.425831845 +0000 UTC m=+203.400058326" Oct 11 10:29:31.443991 master-1 kubenswrapper[4771]: I1011 10:29:31.443919 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-79d5f95f5c-67qps" podStartSLOduration=5.959374821 podStartE2EDuration="10.443899573s" podCreationTimestamp="2025-10-11 10:29:21 +0000 UTC" firstStartedPulling="2025-10-11 10:29:25.909389896 +0000 UTC m=+197.883616337" lastFinishedPulling="2025-10-11 10:29:30.393914648 +0000 UTC m=+202.368141089" observedRunningTime="2025-10-11 10:29:31.441813906 +0000 UTC m=+203.416040347" watchObservedRunningTime="2025-10-11 10:29:31.443899573 +0000 UTC m=+203.418126014" Oct 11 10:29:31.494734 master-1 kubenswrapper[4771]: I1011 10:29:31.494402 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:31.497924 master-1 kubenswrapper[4771]: I1011 10:29:31.497775 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:31.497924 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:31.497924 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:31.497924 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:31.498132 master-1 kubenswrapper[4771]: I1011 10:29:31.497909 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:31.533040 master-1 kubenswrapper[4771]: I1011 10:29:31.532976 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podStartSLOduration=40.100956873 podStartE2EDuration="44.532960908s" podCreationTimestamp="2025-10-11 10:28:47 +0000 UTC" firstStartedPulling="2025-10-11 10:29:25.544419654 +0000 UTC m=+197.518646105" lastFinishedPulling="2025-10-11 10:29:29.976423689 +0000 UTC m=+201.950650140" observedRunningTime="2025-10-11 10:29:31.530615714 +0000 UTC m=+203.504842205" watchObservedRunningTime="2025-10-11 10:29:31.532960908 +0000 UTC m=+203.507187349" Oct 11 10:29:31.565644 master-1 kubenswrapper[4771]: I1011 10:29:31.565602 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-controller-manager/installer-2-master-1"] Oct 11 10:29:31.566175 master-1 kubenswrapper[4771]: I1011 10:29:31.565919 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/installer-2-master-1" podUID="792389a1-400d-4a07-a0a5-e80b2edfd8f1" containerName="installer" containerID="cri-o://1847b9a9f31d4cf6b7fede3d6231e62c7c7aec1680e7c800a880c6ba363a8798" gracePeriod=30 Oct 11 10:29:32.421224 master-1 kubenswrapper[4771]: I1011 10:29:32.421157 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-2-master-1_792389a1-400d-4a07-a0a5-e80b2edfd8f1/installer/0.log" Oct 11 10:29:32.421684 master-1 kubenswrapper[4771]: I1011 10:29:32.421231 4771 generic.go:334] "Generic (PLEG): container finished" podID="792389a1-400d-4a07-a0a5-e80b2edfd8f1" containerID="1847b9a9f31d4cf6b7fede3d6231e62c7c7aec1680e7c800a880c6ba363a8798" exitCode=1 Oct 11 10:29:32.422007 master-1 kubenswrapper[4771]: I1011 10:29:32.421961 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-2-master-1" event={"ID":"792389a1-400d-4a07-a0a5-e80b2edfd8f1","Type":"ContainerDied","Data":"1847b9a9f31d4cf6b7fede3d6231e62c7c7aec1680e7c800a880c6ba363a8798"} Oct 11 10:29:32.499609 master-1 kubenswrapper[4771]: I1011 10:29:32.499072 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:32.499609 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:32.499609 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:32.499609 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:32.499609 master-1 kubenswrapper[4771]: I1011 10:29:32.499133 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:32.621597 master-1 kubenswrapper[4771]: I1011 10:29:32.621572 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-2-master-1_792389a1-400d-4a07-a0a5-e80b2edfd8f1/installer/0.log" Oct 11 10:29:32.621705 master-1 kubenswrapper[4771]: I1011 10:29:32.621635 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:32.759653 master-1 kubenswrapper[4771]: I1011 10:29:32.759263 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kube-api-access\") pod \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " Oct 11 10:29:32.759653 master-1 kubenswrapper[4771]: I1011 10:29:32.759399 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-var-lock\") pod \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " Oct 11 10:29:32.759653 master-1 kubenswrapper[4771]: I1011 10:29:32.759436 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kubelet-dir\") pod \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\" (UID: \"792389a1-400d-4a07-a0a5-e80b2edfd8f1\") " Oct 11 10:29:32.759653 master-1 kubenswrapper[4771]: I1011 10:29:32.759618 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "792389a1-400d-4a07-a0a5-e80b2edfd8f1" (UID: "792389a1-400d-4a07-a0a5-e80b2edfd8f1"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:32.759653 master-1 kubenswrapper[4771]: I1011 10:29:32.759617 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-var-lock" (OuterVolumeSpecName: "var-lock") pod "792389a1-400d-4a07-a0a5-e80b2edfd8f1" (UID: "792389a1-400d-4a07-a0a5-e80b2edfd8f1"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:32.762380 master-1 kubenswrapper[4771]: I1011 10:29:32.762313 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "792389a1-400d-4a07-a0a5-e80b2edfd8f1" (UID: "792389a1-400d-4a07-a0a5-e80b2edfd8f1"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:29:32.860870 master-1 kubenswrapper[4771]: I1011 10:29:32.860798 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:32.860870 master-1 kubenswrapper[4771]: I1011 10:29:32.860834 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:32.860870 master-1 kubenswrapper[4771]: I1011 10:29:32.860845 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/792389a1-400d-4a07-a0a5-e80b2edfd8f1-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:33.427002 master-1 kubenswrapper[4771]: I1011 10:29:33.426921 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6" exitCode=0 Oct 11 10:29:33.427679 master-1 kubenswrapper[4771]: I1011 10:29:33.427009 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerDied","Data":"7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6"} Oct 11 10:29:33.430222 master-1 kubenswrapper[4771]: I1011 10:29:33.430197 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-2-master-1_792389a1-400d-4a07-a0a5-e80b2edfd8f1/installer/0.log" Oct 11 10:29:33.430796 master-1 kubenswrapper[4771]: I1011 10:29:33.430761 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-2-master-1" Oct 11 10:29:33.438239 master-1 kubenswrapper[4771]: I1011 10:29:33.437173 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-2-master-1" event={"ID":"792389a1-400d-4a07-a0a5-e80b2edfd8f1","Type":"ContainerDied","Data":"278d42f198fc93ee50b135376d28ae4eb2fe4bcf6f5f1c9223b4e9e7ffd7be30"} Oct 11 10:29:33.438239 master-1 kubenswrapper[4771]: I1011 10:29:33.437644 4771 scope.go:117] "RemoveContainer" containerID="1847b9a9f31d4cf6b7fede3d6231e62c7c7aec1680e7c800a880c6ba363a8798" Oct 11 10:29:33.483217 master-1 kubenswrapper[4771]: I1011 10:29:33.483152 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-controller-manager/installer-2-master-1"] Oct 11 10:29:33.485437 master-1 kubenswrapper[4771]: I1011 10:29:33.485391 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-controller-manager/installer-2-master-1"] Oct 11 10:29:33.496699 master-1 kubenswrapper[4771]: I1011 10:29:33.496652 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:33.496699 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:33.496699 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:33.496699 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:33.496965 master-1 kubenswrapper[4771]: I1011 10:29:33.496712 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:34.296386 master-1 kubenswrapper[4771]: I1011 10:29:34.296026 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-etcd/etcd-guard-master-1"] Oct 11 10:29:34.296386 master-1 kubenswrapper[4771]: E1011 10:29:34.296188 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="826e1279-bc0d-426e-b6e0-5108268f340e" containerName="installer" Oct 11 10:29:34.296386 master-1 kubenswrapper[4771]: I1011 10:29:34.296203 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="826e1279-bc0d-426e-b6e0-5108268f340e" containerName="installer" Oct 11 10:29:34.296386 master-1 kubenswrapper[4771]: E1011 10:29:34.296213 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="792389a1-400d-4a07-a0a5-e80b2edfd8f1" containerName="installer" Oct 11 10:29:34.296386 master-1 kubenswrapper[4771]: I1011 10:29:34.296221 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="792389a1-400d-4a07-a0a5-e80b2edfd8f1" containerName="installer" Oct 11 10:29:34.296386 master-1 kubenswrapper[4771]: I1011 10:29:34.296296 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="826e1279-bc0d-426e-b6e0-5108268f340e" containerName="installer" Oct 11 10:29:34.296386 master-1 kubenswrapper[4771]: I1011 10:29:34.296307 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="792389a1-400d-4a07-a0a5-e80b2edfd8f1" containerName="installer" Oct 11 10:29:34.297870 master-1 kubenswrapper[4771]: I1011 10:29:34.297297 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:29:34.302595 master-1 kubenswrapper[4771]: I1011 10:29:34.301606 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-etcd"/"kube-root-ca.crt" Oct 11 10:29:34.302595 master-1 kubenswrapper[4771]: I1011 10:29:34.301657 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-etcd"/"openshift-service-ca.crt" Oct 11 10:29:34.304683 master-1 kubenswrapper[4771]: I1011 10:29:34.304662 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/etcd-guard-master-1"] Oct 11 10:29:34.441902 master-1 kubenswrapper[4771]: I1011 10:29:34.441858 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-1-master-1_6534d9db-a553-4c39-bf4a-014a359ee336/installer/0.log" Oct 11 10:29:34.442394 master-1 kubenswrapper[4771]: I1011 10:29:34.441914 4771 generic.go:334] "Generic (PLEG): container finished" podID="6534d9db-a553-4c39-bf4a-014a359ee336" containerID="c9e465db2f016eeb1b9eb6a1701316ad91386e0556613224875082e886221894" exitCode=1 Oct 11 10:29:34.444140 master-1 kubenswrapper[4771]: I1011 10:29:34.444093 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="792389a1-400d-4a07-a0a5-e80b2edfd8f1" path="/var/lib/kubelet/pods/792389a1-400d-4a07-a0a5-e80b2edfd8f1/volumes" Oct 11 10:29:34.444140 master-1 kubenswrapper[4771]: I1011 10:29:34.444116 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55" exitCode=0 Oct 11 10:29:34.444635 master-1 kubenswrapper[4771]: I1011 10:29:34.444590 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-1-master-1" event={"ID":"6534d9db-a553-4c39-bf4a-014a359ee336","Type":"ContainerDied","Data":"c9e465db2f016eeb1b9eb6a1701316ad91386e0556613224875082e886221894"} Oct 11 10:29:34.444635 master-1 kubenswrapper[4771]: I1011 10:29:34.444630 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerDied","Data":"73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55"} Oct 11 10:29:34.482024 master-1 kubenswrapper[4771]: I1011 10:29:34.481909 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wvj\" (UniqueName: \"kubernetes.io/projected/3fc4970d-4f34-4fc6-9791-6218f8e42eb9-kube-api-access-g9wvj\") pod \"etcd-guard-master-1\" (UID: \"3fc4970d-4f34-4fc6-9791-6218f8e42eb9\") " pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:29:34.502087 master-1 kubenswrapper[4771]: I1011 10:29:34.502021 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:34.502087 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:34.502087 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:34.502087 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:34.502302 master-1 kubenswrapper[4771]: I1011 10:29:34.502115 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:34.557084 master-1 kubenswrapper[4771]: I1011 10:29:34.557029 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-1-master-1_6534d9db-a553-4c39-bf4a-014a359ee336/installer/0.log" Oct 11 10:29:34.557230 master-1 kubenswrapper[4771]: I1011 10:29:34.557109 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:34.587444 master-1 kubenswrapper[4771]: I1011 10:29:34.587374 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-var-lock\") pod \"6534d9db-a553-4c39-bf4a-014a359ee336\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " Oct 11 10:29:34.587654 master-1 kubenswrapper[4771]: I1011 10:29:34.587456 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-kubelet-dir\") pod \"6534d9db-a553-4c39-bf4a-014a359ee336\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " Oct 11 10:29:34.587654 master-1 kubenswrapper[4771]: I1011 10:29:34.587497 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/6534d9db-a553-4c39-bf4a-014a359ee336-kube-api-access\") pod \"6534d9db-a553-4c39-bf4a-014a359ee336\" (UID: \"6534d9db-a553-4c39-bf4a-014a359ee336\") " Oct 11 10:29:34.587654 master-1 kubenswrapper[4771]: I1011 10:29:34.587530 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-var-lock" (OuterVolumeSpecName: "var-lock") pod "6534d9db-a553-4c39-bf4a-014a359ee336" (UID: "6534d9db-a553-4c39-bf4a-014a359ee336"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:34.587654 master-1 kubenswrapper[4771]: I1011 10:29:34.587593 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "6534d9db-a553-4c39-bf4a-014a359ee336" (UID: "6534d9db-a553-4c39-bf4a-014a359ee336"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:34.587654 master-1 kubenswrapper[4771]: I1011 10:29:34.587645 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wvj\" (UniqueName: \"kubernetes.io/projected/3fc4970d-4f34-4fc6-9791-6218f8e42eb9-kube-api-access-g9wvj\") pod \"etcd-guard-master-1\" (UID: \"3fc4970d-4f34-4fc6-9791-6218f8e42eb9\") " pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:29:34.587948 master-1 kubenswrapper[4771]: I1011 10:29:34.587745 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:34.587948 master-1 kubenswrapper[4771]: I1011 10:29:34.587766 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/6534d9db-a553-4c39-bf4a-014a359ee336-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:34.591667 master-1 kubenswrapper[4771]: I1011 10:29:34.591557 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6534d9db-a553-4c39-bf4a-014a359ee336-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "6534d9db-a553-4c39-bf4a-014a359ee336" (UID: "6534d9db-a553-4c39-bf4a-014a359ee336"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:29:34.616537 master-1 kubenswrapper[4771]: I1011 10:29:34.616491 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wvj\" (UniqueName: \"kubernetes.io/projected/3fc4970d-4f34-4fc6-9791-6218f8e42eb9-kube-api-access-g9wvj\") pod \"etcd-guard-master-1\" (UID: \"3fc4970d-4f34-4fc6-9791-6218f8e42eb9\") " pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:29:34.627346 master-1 kubenswrapper[4771]: I1011 10:29:34.627310 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:29:34.688531 master-1 kubenswrapper[4771]: I1011 10:29:34.688499 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/6534d9db-a553-4c39-bf4a-014a359ee336-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:35.060116 master-1 kubenswrapper[4771]: I1011 10:29:35.060030 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/etcd-guard-master-1"] Oct 11 10:29:35.068115 master-1 kubenswrapper[4771]: W1011 10:29:35.068077 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc4970d_4f34_4fc6_9791_6218f8e42eb9.slice/crio-7d7d5bb1e54a26f56bba9c02805eb5d544d7ecd2f66f6f9c7a4e1cd7ea203bc0 WatchSource:0}: Error finding container 7d7d5bb1e54a26f56bba9c02805eb5d544d7ecd2f66f6f9c7a4e1cd7ea203bc0: Status 404 returned error can't find the container with id 7d7d5bb1e54a26f56bba9c02805eb5d544d7ecd2f66f6f9c7a4e1cd7ea203bc0 Oct 11 10:29:35.456613 master-1 kubenswrapper[4771]: I1011 10:29:35.456566 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-1-master-1_6534d9db-a553-4c39-bf4a-014a359ee336/installer/0.log" Oct 11 10:29:35.457064 master-1 kubenswrapper[4771]: I1011 10:29:35.456861 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-1-master-1" Oct 11 10:29:35.457561 master-1 kubenswrapper[4771]: I1011 10:29:35.457206 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-1-master-1" event={"ID":"6534d9db-a553-4c39-bf4a-014a359ee336","Type":"ContainerDied","Data":"b5b289645c8dafc708db0dfb37bf1e6882fdc062aac0a46f6f992e36cadc5dc7"} Oct 11 10:29:35.457561 master-1 kubenswrapper[4771]: I1011 10:29:35.457292 4771 scope.go:117] "RemoveContainer" containerID="c9e465db2f016eeb1b9eb6a1701316ad91386e0556613224875082e886221894" Oct 11 10:29:35.465712 master-1 kubenswrapper[4771]: I1011 10:29:35.465659 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05" exitCode=0 Oct 11 10:29:35.466133 master-1 kubenswrapper[4771]: I1011 10:29:35.465794 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerDied","Data":"ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05"} Oct 11 10:29:35.467761 master-1 kubenswrapper[4771]: I1011 10:29:35.467716 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-guard-master-1" event={"ID":"3fc4970d-4f34-4fc6-9791-6218f8e42eb9","Type":"ContainerStarted","Data":"7d7d5bb1e54a26f56bba9c02805eb5d544d7ecd2f66f6f9c7a4e1cd7ea203bc0"} Oct 11 10:29:35.494635 master-1 kubenswrapper[4771]: I1011 10:29:35.494571 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:29:35.496565 master-1 kubenswrapper[4771]: I1011 10:29:35.496530 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:35.496565 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:35.496565 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:35.496565 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:35.496750 master-1 kubenswrapper[4771]: I1011 10:29:35.496572 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:35.506143 master-1 kubenswrapper[4771]: I1011 10:29:35.506099 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-controller-manager/installer-1-master-1"] Oct 11 10:29:35.510935 master-1 kubenswrapper[4771]: I1011 10:29:35.510888 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-controller-manager/installer-1-master-1"] Oct 11 10:29:36.442339 master-1 kubenswrapper[4771]: I1011 10:29:36.442264 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6534d9db-a553-4c39-bf4a-014a359ee336" path="/var/lib/kubelet/pods/6534d9db-a553-4c39-bf4a-014a359ee336/volumes" Oct 11 10:29:36.474648 master-1 kubenswrapper[4771]: I1011 10:29:36.473562 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-guard-master-1" event={"ID":"3fc4970d-4f34-4fc6-9791-6218f8e42eb9","Type":"ContainerStarted","Data":"d9979a6eb84532ea8c8fdf3474fb9576ae4ba27a4e7d93fc9eeeadab89ea349f"} Oct 11 10:29:36.474648 master-1 kubenswrapper[4771]: I1011 10:29:36.474599 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:29:36.476128 master-1 kubenswrapper[4771]: I1011 10:29:36.475623 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:29:36.476128 master-1 kubenswrapper[4771]: I1011 10:29:36.475685 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:29:36.478077 master-1 kubenswrapper[4771]: I1011 10:29:36.478033 4771 generic.go:334] "Generic (PLEG): container finished" podID="868ea5b9-b62a-4683-82c9-760de94ef155" containerID="081fde9dac0d8c6f0177a9a06139a4e92fb38ea47b03713cf7f04ea063469f84" exitCode=0 Oct 11 10:29:36.478208 master-1 kubenswrapper[4771]: I1011 10:29:36.478101 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" event={"ID":"868ea5b9-b62a-4683-82c9-760de94ef155","Type":"ContainerDied","Data":"081fde9dac0d8c6f0177a9a06139a4e92fb38ea47b03713cf7f04ea063469f84"} Oct 11 10:29:36.478446 master-1 kubenswrapper[4771]: I1011 10:29:36.478409 4771 scope.go:117] "RemoveContainer" containerID="081fde9dac0d8c6f0177a9a06139a4e92fb38ea47b03713cf7f04ea063469f84" Oct 11 10:29:36.481568 master-1 kubenswrapper[4771]: I1011 10:29:36.481537 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerStarted","Data":"2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8"} Oct 11 10:29:36.493427 master-1 kubenswrapper[4771]: I1011 10:29:36.493334 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-etcd/etcd-guard-master-1" podStartSLOduration=2.49331896 podStartE2EDuration="2.49331896s" podCreationTimestamp="2025-10-11 10:29:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:36.492534218 +0000 UTC m=+208.466760689" watchObservedRunningTime="2025-10-11 10:29:36.49331896 +0000 UTC m=+208.467545421" Oct 11 10:29:36.497771 master-1 kubenswrapper[4771]: I1011 10:29:36.497733 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:36.497771 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:36.497771 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:36.497771 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:36.498087 master-1 kubenswrapper[4771]: I1011 10:29:36.497783 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:37.486043 master-1 kubenswrapper[4771]: I1011 10:29:37.485867 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:29:37.486043 master-1 kubenswrapper[4771]: I1011 10:29:37.485949 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:29:37.497188 master-1 kubenswrapper[4771]: I1011 10:29:37.497063 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:37.497188 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:37.497188 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:37.497188 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:37.497188 master-1 kubenswrapper[4771]: I1011 10:29:37.497123 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:38.488981 master-1 kubenswrapper[4771]: I1011 10:29:38.488832 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:29:38.488981 master-1 kubenswrapper[4771]: I1011 10:29:38.488916 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:29:38.497653 master-1 kubenswrapper[4771]: I1011 10:29:38.497544 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:38.497653 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:38.497653 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:38.497653 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:38.497653 master-1 kubenswrapper[4771]: I1011 10:29:38.497645 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:38.898176 master-1 kubenswrapper[4771]: I1011 10:29:38.898129 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/etcd-guard-master-1"] Oct 11 10:29:39.335509 master-1 kubenswrapper[4771]: I1011 10:29:39.335458 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/installer-3-master-1"] Oct 11 10:29:39.335729 master-1 kubenswrapper[4771]: E1011 10:29:39.335625 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="6534d9db-a553-4c39-bf4a-014a359ee336" containerName="installer" Oct 11 10:29:39.335729 master-1 kubenswrapper[4771]: I1011 10:29:39.335642 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="6534d9db-a553-4c39-bf4a-014a359ee336" containerName="installer" Oct 11 10:29:39.335729 master-1 kubenswrapper[4771]: I1011 10:29:39.335713 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="6534d9db-a553-4c39-bf4a-014a359ee336" containerName="installer" Oct 11 10:29:39.336096 master-1 kubenswrapper[4771]: I1011 10:29:39.336073 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.340763 master-1 kubenswrapper[4771]: I1011 10:29:39.340727 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-controller-manager"/"kube-root-ca.crt" Oct 11 10:29:39.342221 master-1 kubenswrapper[4771]: I1011 10:29:39.342182 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-3-master-1"] Oct 11 10:29:39.442185 master-1 kubenswrapper[4771]: I1011 10:29:39.442123 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kube-api-access\") pod \"installer-3-master-1\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.442636 master-1 kubenswrapper[4771]: I1011 10:29:39.442208 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kubelet-dir\") pod \"installer-3-master-1\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.443660 master-1 kubenswrapper[4771]: I1011 10:29:39.443011 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-var-lock\") pod \"installer-3-master-1\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.491841 master-1 kubenswrapper[4771]: I1011 10:29:39.491759 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:29:39.491841 master-1 kubenswrapper[4771]: I1011 10:29:39.491828 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:29:39.496298 master-1 kubenswrapper[4771]: I1011 10:29:39.496255 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:39.496298 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:39.496298 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:39.496298 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:39.496501 master-1 kubenswrapper[4771]: I1011 10:29:39.496313 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:39.550215 master-1 kubenswrapper[4771]: I1011 10:29:39.550131 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kube-api-access\") pod \"installer-3-master-1\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.550215 master-1 kubenswrapper[4771]: I1011 10:29:39.550214 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kubelet-dir\") pod \"installer-3-master-1\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.550491 master-1 kubenswrapper[4771]: I1011 10:29:39.550275 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-var-lock\") pod \"installer-3-master-1\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.550537 master-1 kubenswrapper[4771]: I1011 10:29:39.550515 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-var-lock\") pod \"installer-3-master-1\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.550603 master-1 kubenswrapper[4771]: I1011 10:29:39.550573 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kubelet-dir\") pod \"installer-3-master-1\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.578484 master-1 kubenswrapper[4771]: I1011 10:29:39.578404 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kube-api-access\") pod \"installer-3-master-1\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:39.628986 master-1 kubenswrapper[4771]: I1011 10:29:39.628784 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:29:39.628986 master-1 kubenswrapper[4771]: I1011 10:29:39.628885 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:29:39.651382 master-1 kubenswrapper[4771]: I1011 10:29:39.651258 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:29:40.496591 master-1 kubenswrapper[4771]: I1011 10:29:40.496508 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:40.496591 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:40.496591 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:40.496591 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:40.497237 master-1 kubenswrapper[4771]: I1011 10:29:40.496596 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:41.497425 master-1 kubenswrapper[4771]: I1011 10:29:41.497322 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:41.497425 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:41.497425 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:41.497425 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:41.497425 master-1 kubenswrapper[4771]: I1011 10:29:41.497408 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:42.496641 master-1 kubenswrapper[4771]: I1011 10:29:42.496593 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:42.496641 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:42.496641 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:42.496641 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:42.496919 master-1 kubenswrapper[4771]: I1011 10:29:42.496654 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:43.499835 master-1 kubenswrapper[4771]: I1011 10:29:43.499782 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:43.499835 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:43.499835 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:43.499835 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:43.500254 master-1 kubenswrapper[4771]: I1011 10:29:43.499833 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:43.512600 master-1 kubenswrapper[4771]: I1011 10:29:43.512559 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-gwwz9" event={"ID":"0b7d1d62-0062-47cd-a963-63893777198e","Type":"ContainerStarted","Data":"d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80"} Oct 11 10:29:43.515342 master-1 kubenswrapper[4771]: I1011 10:29:43.515301 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-g8tm6" event={"ID":"38131fcf-d407-4ba3-b7bf-471586bab887","Type":"ContainerStarted","Data":"10eecae7180584a993b9109e41de9729732ec8af959166bad8fe7ba33a08f83b"} Oct 11 10:29:43.517009 master-1 kubenswrapper[4771]: I1011 10:29:43.516988 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/network-operator-854f54f8c9-hw5fc" event={"ID":"868ea5b9-b62a-4683-82c9-760de94ef155","Type":"ContainerStarted","Data":"082eee2a8f515d9f2a8d8b7d1ce478df6413dccc9907ab7bcad6ffb296b971cc"} Oct 11 10:29:43.520251 master-1 kubenswrapper[4771]: I1011 10:29:43.520227 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerStarted","Data":"d00f571fb5251acb052a97b0ee5169046519d14d8990c98b7ea440fa842ffd37"} Oct 11 10:29:43.645836 master-1 kubenswrapper[4771]: I1011 10:29:43.645769 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-3-master-1"] Oct 11 10:29:43.734224 master-1 kubenswrapper[4771]: W1011 10:29:43.734161 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod776c3745_2f4c_4a78_b1cd_77a7a1532df3.slice/crio-ccc3bd0dec107a1860fda6d334afbd26f993ae533a829286bea5468aeefd8bf7 WatchSource:0}: Error finding container ccc3bd0dec107a1860fda6d334afbd26f993ae533a829286bea5468aeefd8bf7: Status 404 returned error can't find the container with id ccc3bd0dec107a1860fda6d334afbd26f993ae533a829286bea5468aeefd8bf7 Oct 11 10:29:44.104088 master-1 kubenswrapper[4771]: I1011 10:29:44.104027 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-9d7j4"] Oct 11 10:29:44.104701 master-1 kubenswrapper[4771]: I1011 10:29:44.104673 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.107977 master-1 kubenswrapper[4771]: I1011 10:29:44.107943 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-multus"/"cni-sysctl-allowlist" Oct 11 10:29:44.203644 master-1 kubenswrapper[4771]: I1011 10:29:44.203512 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgsk\" (UniqueName: \"kubernetes.io/projected/806cd59c-056a-4fb4-a3b4-cb716c01cdea-kube-api-access-tfgsk\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.203644 master-1 kubenswrapper[4771]: I1011 10:29:44.203588 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/806cd59c-056a-4fb4-a3b4-cb716c01cdea-ready\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.203871 master-1 kubenswrapper[4771]: I1011 10:29:44.203666 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/806cd59c-056a-4fb4-a3b4-cb716c01cdea-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.203871 master-1 kubenswrapper[4771]: I1011 10:29:44.203808 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/806cd59c-056a-4fb4-a3b4-cb716c01cdea-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.305060 master-1 kubenswrapper[4771]: I1011 10:29:44.305011 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgsk\" (UniqueName: \"kubernetes.io/projected/806cd59c-056a-4fb4-a3b4-cb716c01cdea-kube-api-access-tfgsk\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.305060 master-1 kubenswrapper[4771]: I1011 10:29:44.305057 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/806cd59c-056a-4fb4-a3b4-cb716c01cdea-ready\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.305536 master-1 kubenswrapper[4771]: I1011 10:29:44.305093 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/806cd59c-056a-4fb4-a3b4-cb716c01cdea-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.305536 master-1 kubenswrapper[4771]: I1011 10:29:44.305126 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/806cd59c-056a-4fb4-a3b4-cb716c01cdea-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.305536 master-1 kubenswrapper[4771]: I1011 10:29:44.305235 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/806cd59c-056a-4fb4-a3b4-cb716c01cdea-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.305787 master-1 kubenswrapper[4771]: I1011 10:29:44.305598 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/806cd59c-056a-4fb4-a3b4-cb716c01cdea-ready\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.306948 master-1 kubenswrapper[4771]: I1011 10:29:44.306872 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/806cd59c-056a-4fb4-a3b4-cb716c01cdea-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.324555 master-1 kubenswrapper[4771]: I1011 10:29:44.324485 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgsk\" (UniqueName: \"kubernetes.io/projected/806cd59c-056a-4fb4-a3b4-cb716c01cdea-kube-api-access-tfgsk\") pod \"cni-sysctl-allowlist-ds-9d7j4\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.448781 master-1 kubenswrapper[4771]: I1011 10:29:44.448684 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:44.467144 master-1 kubenswrapper[4771]: W1011 10:29:44.466859 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806cd59c_056a_4fb4_a3b4_cb716c01cdea.slice/crio-7147f14021bd8181c058d8c3ce2203cdae664d32eab5196f21ee167281d79073 WatchSource:0}: Error finding container 7147f14021bd8181c058d8c3ce2203cdae664d32eab5196f21ee167281d79073: Status 404 returned error can't find the container with id 7147f14021bd8181c058d8c3ce2203cdae664d32eab5196f21ee167281d79073 Oct 11 10:29:44.497494 master-1 kubenswrapper[4771]: I1011 10:29:44.497437 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:44.497494 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:44.497494 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:44.497494 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:44.497862 master-1 kubenswrapper[4771]: I1011 10:29:44.497526 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:44.536188 master-1 kubenswrapper[4771]: I1011 10:29:44.536094 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-3-master-1" event={"ID":"776c3745-2f4c-4a78-b1cd-77a7a1532df3","Type":"ContainerStarted","Data":"2a2c47f6b163a67c15dfe1ca6c1ec25571de95f1ae3f653d4b9ded6b99ad45a9"} Oct 11 10:29:44.536188 master-1 kubenswrapper[4771]: I1011 10:29:44.536182 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-3-master-1" event={"ID":"776c3745-2f4c-4a78-b1cd-77a7a1532df3","Type":"ContainerStarted","Data":"ccc3bd0dec107a1860fda6d334afbd26f993ae533a829286bea5468aeefd8bf7"} Oct 11 10:29:44.542321 master-1 kubenswrapper[4771]: I1011 10:29:44.542277 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerStarted","Data":"958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0"} Oct 11 10:29:44.542438 master-1 kubenswrapper[4771]: I1011 10:29:44.542334 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerStarted","Data":"cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711"} Oct 11 10:29:44.542438 master-1 kubenswrapper[4771]: I1011 10:29:44.542410 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerStarted","Data":"4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0"} Oct 11 10:29:44.546336 master-1 kubenswrapper[4771]: I1011 10:29:44.546290 4771 generic.go:334] "Generic (PLEG): container finished" podID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerID="4bde2f0bff6002ac88c69a20de25c24e27ed2402f74ddf6b6f429bda18e25de4" exitCode=0 Oct 11 10:29:44.546467 master-1 kubenswrapper[4771]: I1011 10:29:44.546432 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-xkrc6" event={"ID":"26005893-ecd8-4acb-8417-71a97ed97cbe","Type":"ContainerDied","Data":"4bde2f0bff6002ac88c69a20de25c24e27ed2402f74ddf6b6f429bda18e25de4"} Oct 11 10:29:44.550590 master-1 kubenswrapper[4771]: I1011 10:29:44.550534 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b7d1d62-0062-47cd-a963-63893777198e" containerID="d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80" exitCode=0 Oct 11 10:29:44.550705 master-1 kubenswrapper[4771]: I1011 10:29:44.550603 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-gwwz9" event={"ID":"0b7d1d62-0062-47cd-a963-63893777198e","Type":"ContainerDied","Data":"d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80"} Oct 11 10:29:44.556692 master-1 kubenswrapper[4771]: I1011 10:29:44.556642 4771 generic.go:334] "Generic (PLEG): container finished" podID="38131fcf-d407-4ba3-b7bf-471586bab887" containerID="10eecae7180584a993b9109e41de9729732ec8af959166bad8fe7ba33a08f83b" exitCode=0 Oct 11 10:29:44.556823 master-1 kubenswrapper[4771]: I1011 10:29:44.556774 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-g8tm6" event={"ID":"38131fcf-d407-4ba3-b7bf-471586bab887","Type":"ContainerDied","Data":"10eecae7180584a993b9109e41de9729732ec8af959166bad8fe7ba33a08f83b"} Oct 11 10:29:44.559192 master-1 kubenswrapper[4771]: I1011 10:29:44.559139 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" event={"ID":"806cd59c-056a-4fb4-a3b4-cb716c01cdea","Type":"ContainerStarted","Data":"7147f14021bd8181c058d8c3ce2203cdae664d32eab5196f21ee167281d79073"} Oct 11 10:29:44.560623 master-1 kubenswrapper[4771]: I1011 10:29:44.560540 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/installer-3-master-1" podStartSLOduration=5.560514138 podStartE2EDuration="5.560514138s" podCreationTimestamp="2025-10-11 10:29:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:44.556816216 +0000 UTC m=+216.531042717" watchObservedRunningTime="2025-10-11 10:29:44.560514138 +0000 UTC m=+216.534740619" Oct 11 10:29:44.566670 master-1 kubenswrapper[4771]: I1011 10:29:44.566612 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dvv69"] Oct 11 10:29:44.567842 master-1 kubenswrapper[4771]: I1011 10:29:44.567803 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.570014 master-1 kubenswrapper[4771]: I1011 10:29:44.569923 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs"] Oct 11 10:29:44.571278 master-1 kubenswrapper[4771]: I1011 10:29:44.571197 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.574709 master-1 kubenswrapper[4771]: I1011 10:29:44.574661 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"metrics-client-ca" Oct 11 10:29:44.577290 master-1 kubenswrapper[4771]: I1011 10:29:44.577217 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"kube-root-ca.crt" Oct 11 10:29:44.577962 master-1 kubenswrapper[4771]: I1011 10:29:44.577913 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"openshift-service-ca.crt" Oct 11 10:29:44.578367 master-1 kubenswrapper[4771]: I1011 10:29:44.578225 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"openshift-state-metrics-tls" Oct 11 10:29:44.580199 master-1 kubenswrapper[4771]: I1011 10:29:44.580129 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"openshift-state-metrics-kube-rbac-proxy-config" Oct 11 10:29:44.580374 master-1 kubenswrapper[4771]: I1011 10:29:44.580292 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"node-exporter-tls" Oct 11 10:29:44.580883 master-1 kubenswrapper[4771]: I1011 10:29:44.580855 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"node-exporter-kube-rbac-proxy-config" Oct 11 10:29:44.597651 master-1 kubenswrapper[4771]: I1011 10:29:44.597598 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs"] Oct 11 10:29:44.620652 master-1 kubenswrapper[4771]: I1011 10:29:44.612230 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-root\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.620944 master-1 kubenswrapper[4771]: I1011 10:29:44.614614 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-etcd/etcd-master-1" podStartSLOduration=17.614576139 podStartE2EDuration="17.614576139s" podCreationTimestamp="2025-10-11 10:29:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:44.613928011 +0000 UTC m=+216.588154512" watchObservedRunningTime="2025-10-11 10:29:44.614576139 +0000 UTC m=+216.588802630" Oct 11 10:29:44.620944 master-1 kubenswrapper[4771]: I1011 10:29:44.620792 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-wtmp\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.620944 master-1 kubenswrapper[4771]: I1011 10:29:44.620895 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcz25\" (UniqueName: \"kubernetes.io/projected/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-kube-api-access-qcz25\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.621103 master-1 kubenswrapper[4771]: I1011 10:29:44.620941 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4893176f-942c-49bf-aaab-9c238ecdaaa7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.621103 master-1 kubenswrapper[4771]: I1011 10:29:44.621009 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wchwm\" (UniqueName: \"kubernetes.io/projected/4893176f-942c-49bf-aaab-9c238ecdaaa7-kube-api-access-wchwm\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.621103 master-1 kubenswrapper[4771]: I1011 10:29:44.621040 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-metrics-client-ca\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.621232 master-1 kubenswrapper[4771]: I1011 10:29:44.621147 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-textfile\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.621232 master-1 kubenswrapper[4771]: I1011 10:29:44.621181 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-tls\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.621317 master-1 kubenswrapper[4771]: I1011 10:29:44.621293 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-sys\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.621380 master-1 kubenswrapper[4771]: I1011 10:29:44.621338 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.622304 master-1 kubenswrapper[4771]: I1011 10:29:44.621455 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4893176f-942c-49bf-aaab-9c238ecdaaa7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.622304 master-1 kubenswrapper[4771]: I1011 10:29:44.621502 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4893176f-942c-49bf-aaab-9c238ecdaaa7-metrics-client-ca\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722220 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-qcz25\" (UniqueName: \"kubernetes.io/projected/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-kube-api-access-qcz25\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722290 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4893176f-942c-49bf-aaab-9c238ecdaaa7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722329 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-wchwm\" (UniqueName: \"kubernetes.io/projected/4893176f-942c-49bf-aaab-9c238ecdaaa7-kube-api-access-wchwm\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722392 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-metrics-client-ca\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722458 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-tls\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722493 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-textfile\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722545 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-sys\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722578 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722650 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4893176f-942c-49bf-aaab-9c238ecdaaa7-metrics-client-ca\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722682 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4893176f-942c-49bf-aaab-9c238ecdaaa7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722735 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-root\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.722764 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-wtmp\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.723018 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-wtmp\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.723079 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-root\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.723300 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-textfile\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.723644 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-metrics-client-ca\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.723726 master-1 kubenswrapper[4771]: I1011 10:29:44.723642 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-sys\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.724750 master-1 kubenswrapper[4771]: I1011 10:29:44.724709 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4893176f-942c-49bf-aaab-9c238ecdaaa7-metrics-client-ca\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.726741 master-1 kubenswrapper[4771]: I1011 10:29:44.726701 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4893176f-942c-49bf-aaab-9c238ecdaaa7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.727586 master-1 kubenswrapper[4771]: I1011 10:29:44.727540 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-tls\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.728514 master-1 kubenswrapper[4771]: I1011 10:29:44.728472 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.729903 master-1 kubenswrapper[4771]: I1011 10:29:44.729863 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4893176f-942c-49bf-aaab-9c238ecdaaa7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.749216 master-1 kubenswrapper[4771]: I1011 10:29:44.749175 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcz25\" (UniqueName: \"kubernetes.io/projected/bb58d9ff-af20-40c4-9dfe-c9c10fb5c410-kube-api-access-qcz25\") pod \"node-exporter-dvv69\" (UID: \"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410\") " pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.757258 master-1 kubenswrapper[4771]: I1011 10:29:44.757198 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-wchwm\" (UniqueName: \"kubernetes.io/projected/4893176f-942c-49bf-aaab-9c238ecdaaa7-kube-api-access-wchwm\") pod \"openshift-state-metrics-56d8dcb55c-xgtjs\" (UID: \"4893176f-942c-49bf-aaab-9c238ecdaaa7\") " pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.895866 master-1 kubenswrapper[4771]: I1011 10:29:44.895755 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dvv69" Oct 11 10:29:44.910196 master-1 kubenswrapper[4771]: I1011 10:29:44.910140 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" Oct 11 10:29:44.910409 master-1 kubenswrapper[4771]: W1011 10:29:44.910340 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb58d9ff_af20_40c4_9dfe_c9c10fb5c410.slice/crio-58922d7545ec86f39b36fd35d48be93f0e0a80f1b4264052cd2d4e50edcd3ffd WatchSource:0}: Error finding container 58922d7545ec86f39b36fd35d48be93f0e0a80f1b4264052cd2d4e50edcd3ffd: Status 404 returned error can't find the container with id 58922d7545ec86f39b36fd35d48be93f0e0a80f1b4264052cd2d4e50edcd3ffd Oct 11 10:29:45.312249 master-1 kubenswrapper[4771]: I1011 10:29:45.312184 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs"] Oct 11 10:29:45.318178 master-1 kubenswrapper[4771]: W1011 10:29:45.318114 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4893176f_942c_49bf_aaab_9c238ecdaaa7.slice/crio-339368d8966a6c55474373995ab7421c35827ed42bcee77b3b7f7b2f23b12cf7 WatchSource:0}: Error finding container 339368d8966a6c55474373995ab7421c35827ed42bcee77b3b7f7b2f23b12cf7: Status 404 returned error can't find the container with id 339368d8966a6c55474373995ab7421c35827ed42bcee77b3b7f7b2f23b12cf7 Oct 11 10:29:45.496785 master-1 kubenswrapper[4771]: I1011 10:29:45.496744 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:45.496785 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:45.496785 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:45.496785 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:45.497073 master-1 kubenswrapper[4771]: I1011 10:29:45.496804 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:45.567510 master-1 kubenswrapper[4771]: I1011 10:29:45.567331 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-xkrc6" event={"ID":"26005893-ecd8-4acb-8417-71a97ed97cbe","Type":"ContainerStarted","Data":"9f359af209588aa409904f71581bb63e20e019ac6f684b2bb1874bdc33d16458"} Oct 11 10:29:45.576559 master-1 kubenswrapper[4771]: I1011 10:29:45.576510 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-gwwz9" event={"ID":"0b7d1d62-0062-47cd-a963-63893777198e","Type":"ContainerStarted","Data":"f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20"} Oct 11 10:29:45.581220 master-1 kubenswrapper[4771]: I1011 10:29:45.581171 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dvv69" event={"ID":"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410","Type":"ContainerStarted","Data":"58922d7545ec86f39b36fd35d48be93f0e0a80f1b4264052cd2d4e50edcd3ffd"} Oct 11 10:29:45.587234 master-1 kubenswrapper[4771]: I1011 10:29:45.584441 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" event={"ID":"4893176f-942c-49bf-aaab-9c238ecdaaa7","Type":"ContainerStarted","Data":"d4c6df9cc8fee2566997beb74041b60350024094905c50b4f5913430e02a3c29"} Oct 11 10:29:45.587234 master-1 kubenswrapper[4771]: I1011 10:29:45.584493 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" event={"ID":"4893176f-942c-49bf-aaab-9c238ecdaaa7","Type":"ContainerStarted","Data":"4f36dcfa6cd4c190aa3fd44b1c6524338d0e8c1c9598feb3a67a82b66a19a482"} Oct 11 10:29:45.587234 master-1 kubenswrapper[4771]: I1011 10:29:45.584508 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" event={"ID":"4893176f-942c-49bf-aaab-9c238ecdaaa7","Type":"ContainerStarted","Data":"339368d8966a6c55474373995ab7421c35827ed42bcee77b3b7f7b2f23b12cf7"} Oct 11 10:29:45.587234 master-1 kubenswrapper[4771]: I1011 10:29:45.585206 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-marketplace-xkrc6" podStartSLOduration=1.737669662 podStartE2EDuration="23.585194776s" podCreationTimestamp="2025-10-11 10:29:22 +0000 UTC" firstStartedPulling="2025-10-11 10:29:23.103742516 +0000 UTC m=+195.077968997" lastFinishedPulling="2025-10-11 10:29:44.95126767 +0000 UTC m=+216.925494111" observedRunningTime="2025-10-11 10:29:45.583831318 +0000 UTC m=+217.558057769" watchObservedRunningTime="2025-10-11 10:29:45.585194776 +0000 UTC m=+217.559421227" Oct 11 10:29:45.587234 master-1 kubenswrapper[4771]: I1011 10:29:45.586782 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-g8tm6" event={"ID":"38131fcf-d407-4ba3-b7bf-471586bab887","Type":"ContainerStarted","Data":"c5ddefdc367347ae7e3aa6121d147be1b4ebca7be06e0180a8a6603ea9ef59cd"} Oct 11 10:29:45.595736 master-1 kubenswrapper[4771]: I1011 10:29:45.595681 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" event={"ID":"806cd59c-056a-4fb4-a3b4-cb716c01cdea","Type":"ContainerStarted","Data":"5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32"} Oct 11 10:29:45.623382 master-1 kubenswrapper[4771]: I1011 10:29:45.623266 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/community-operators-gwwz9" podStartSLOduration=1.756663513 podStartE2EDuration="24.623245985s" podCreationTimestamp="2025-10-11 10:29:21 +0000 UTC" firstStartedPulling="2025-10-11 10:29:22.119285732 +0000 UTC m=+194.093512173" lastFinishedPulling="2025-10-11 10:29:44.985868204 +0000 UTC m=+216.960094645" observedRunningTime="2025-10-11 10:29:45.607792649 +0000 UTC m=+217.582019100" watchObservedRunningTime="2025-10-11 10:29:45.623245985 +0000 UTC m=+217.597472426" Oct 11 10:29:45.639054 master-1 kubenswrapper[4771]: I1011 10:29:45.638853 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-operators-g8tm6" podStartSLOduration=1.89407783 podStartE2EDuration="22.638825634s" podCreationTimestamp="2025-10-11 10:29:23 +0000 UTC" firstStartedPulling="2025-10-11 10:29:24.268890636 +0000 UTC m=+196.243117087" lastFinishedPulling="2025-10-11 10:29:45.01363844 +0000 UTC m=+216.987864891" observedRunningTime="2025-10-11 10:29:45.624383936 +0000 UTC m=+217.598610387" watchObservedRunningTime="2025-10-11 10:29:45.638825634 +0000 UTC m=+217.613052075" Oct 11 10:29:45.639623 master-1 kubenswrapper[4771]: I1011 10:29:45.639574 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" podStartSLOduration=1.639563725 podStartE2EDuration="1.639563725s" podCreationTimestamp="2025-10-11 10:29:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:29:45.638706621 +0000 UTC m=+217.612933082" watchObservedRunningTime="2025-10-11 10:29:45.639563725 +0000 UTC m=+217.613790176" Oct 11 10:29:46.496491 master-1 kubenswrapper[4771]: I1011 10:29:46.496404 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:46.496491 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:46.496491 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:46.496491 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:46.496925 master-1 kubenswrapper[4771]: I1011 10:29:46.496498 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:46.600874 master-1 kubenswrapper[4771]: I1011 10:29:46.600662 4771 generic.go:334] "Generic (PLEG): container finished" podID="bb58d9ff-af20-40c4-9dfe-c9c10fb5c410" containerID="40c63db4dbb96bdb2c0fd427a80f8837a81a5b41a83bc4f90d570adbaaf25d38" exitCode=0 Oct 11 10:29:46.600874 master-1 kubenswrapper[4771]: I1011 10:29:46.600721 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dvv69" event={"ID":"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410","Type":"ContainerDied","Data":"40c63db4dbb96bdb2c0fd427a80f8837a81a5b41a83bc4f90d570adbaaf25d38"} Oct 11 10:29:46.601681 master-1 kubenswrapper[4771]: I1011 10:29:46.601642 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:46.622565 master-1 kubenswrapper[4771]: I1011 10:29:46.622233 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:29:46.860661 master-1 kubenswrapper[4771]: I1011 10:29:46.860595 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:29:46.863886 master-1 kubenswrapper[4771]: I1011 10:29:46.863829 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:29:46.910894 master-1 kubenswrapper[4771]: I1011 10:29:46.910825 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:29:46.953018 master-1 kubenswrapper[4771]: I1011 10:29:46.952961 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-resource-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"89fad8183e18ab3ad0c46d272335e5f8\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:29:46.953150 master-1 kubenswrapper[4771]: I1011 10:29:46.953119 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-cert-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"89fad8183e18ab3ad0c46d272335e5f8\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:29:47.055280 master-1 kubenswrapper[4771]: I1011 10:29:47.055201 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-cert-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"89fad8183e18ab3ad0c46d272335e5f8\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:29:47.055526 master-1 kubenswrapper[4771]: I1011 10:29:47.055286 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-cert-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"89fad8183e18ab3ad0c46d272335e5f8\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:29:47.055526 master-1 kubenswrapper[4771]: I1011 10:29:47.055444 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-resource-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"89fad8183e18ab3ad0c46d272335e5f8\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:29:47.055802 master-1 kubenswrapper[4771]: I1011 10:29:47.055569 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-resource-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"89fad8183e18ab3ad0c46d272335e5f8\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:29:47.111395 master-1 kubenswrapper[4771]: I1011 10:29:47.111200 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-9d7j4"] Oct 11 10:29:47.208023 master-1 kubenswrapper[4771]: I1011 10:29:47.207963 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:29:47.498950 master-1 kubenswrapper[4771]: I1011 10:29:47.498825 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:47.498950 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:47.498950 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:47.498950 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:47.498950 master-1 kubenswrapper[4771]: I1011 10:29:47.498883 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:47.616554 master-1 kubenswrapper[4771]: I1011 10:29:47.616464 4771 generic.go:334] "Generic (PLEG): container finished" podID="7662f87a-13ba-439c-b386-05e68284803c" containerID="6597ee1a813020ee9e9d9c3bc4ac9547370cdcefee548bc443d67590ef76026d" exitCode=0 Oct 11 10:29:47.616554 master-1 kubenswrapper[4771]: I1011 10:29:47.616550 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-4-master-1" event={"ID":"7662f87a-13ba-439c-b386-05e68284803c","Type":"ContainerDied","Data":"6597ee1a813020ee9e9d9c3bc4ac9547370cdcefee548bc443d67590ef76026d"} Oct 11 10:29:47.618029 master-1 kubenswrapper[4771]: I1011 10:29:47.617987 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"89fad8183e18ab3ad0c46d272335e5f8","Type":"ContainerStarted","Data":"bafca73396f947e9fa263ed96b26d1a45ed0144ffb97a2f796fec9628cf617b5"} Oct 11 10:29:47.621801 master-1 kubenswrapper[4771]: I1011 10:29:47.621731 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dvv69" event={"ID":"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410","Type":"ContainerStarted","Data":"24513fb2040a9a28a68a9c8836d96b175f053d07650962edd7c9621aef3f50f3"} Oct 11 10:29:47.621801 master-1 kubenswrapper[4771]: I1011 10:29:47.621777 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dvv69" event={"ID":"bb58d9ff-af20-40c4-9dfe-c9c10fb5c410","Type":"ContainerStarted","Data":"91ae22454328a412be0a05f1050fb49a523649b290c10707524ee5a3197cab20"} Oct 11 10:29:47.624875 master-1 kubenswrapper[4771]: I1011 10:29:47.624830 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" event={"ID":"4893176f-942c-49bf-aaab-9c238ecdaaa7","Type":"ContainerStarted","Data":"0afa205339a7ec15254554c90558c4fb34abcdc63c5bc2d3cadaf2ec152e8a3d"} Oct 11 10:29:47.668487 master-1 kubenswrapper[4771]: I1011 10:29:47.668384 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-56d8dcb55c-xgtjs" podStartSLOduration=2.375374159 podStartE2EDuration="3.668340372s" podCreationTimestamp="2025-10-11 10:29:44 +0000 UTC" firstStartedPulling="2025-10-11 10:29:45.528203345 +0000 UTC m=+217.502429786" lastFinishedPulling="2025-10-11 10:29:46.821169558 +0000 UTC m=+218.795395999" observedRunningTime="2025-10-11 10:29:47.66465646 +0000 UTC m=+219.638882951" watchObservedRunningTime="2025-10-11 10:29:47.668340372 +0000 UTC m=+219.642566833" Oct 11 10:29:47.680194 master-1 kubenswrapper[4771]: I1011 10:29:47.680086 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dvv69" podStartSLOduration=2.511971883 podStartE2EDuration="3.680061114s" podCreationTimestamp="2025-10-11 10:29:44 +0000 UTC" firstStartedPulling="2025-10-11 10:29:44.915424832 +0000 UTC m=+216.889651293" lastFinishedPulling="2025-10-11 10:29:46.083514083 +0000 UTC m=+218.057740524" observedRunningTime="2025-10-11 10:29:47.679172129 +0000 UTC m=+219.653398620" watchObservedRunningTime="2025-10-11 10:29:47.680061114 +0000 UTC m=+219.654287585" Oct 11 10:29:47.772886 master-1 kubenswrapper[4771]: I1011 10:29:47.772716 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:29:48.272298 master-1 kubenswrapper[4771]: I1011 10:29:48.272214 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-etcd/etcd-master-1" Oct 11 10:29:48.272298 master-1 kubenswrapper[4771]: I1011 10:29:48.272294 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-etcd/etcd-master-1" Oct 11 10:29:48.498290 master-1 kubenswrapper[4771]: I1011 10:29:48.498089 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:48.498290 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:48.498290 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:48.498290 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:48.498740 master-1 kubenswrapper[4771]: I1011 10:29:48.498275 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:48.634954 master-1 kubenswrapper[4771]: I1011 10:29:48.634874 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/0.log" Oct 11 10:29:48.636852 master-1 kubenswrapper[4771]: I1011 10:29:48.636793 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="d00f571fb5251acb052a97b0ee5169046519d14d8990c98b7ea440fa842ffd37" exitCode=1 Oct 11 10:29:48.637019 master-1 kubenswrapper[4771]: I1011 10:29:48.636961 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerDied","Data":"d00f571fb5251acb052a97b0ee5169046519d14d8990c98b7ea440fa842ffd37"} Oct 11 10:29:48.637146 master-1 kubenswrapper[4771]: I1011 10:29:48.637086 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" podUID="806cd59c-056a-4fb4-a3b4-cb716c01cdea" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" gracePeriod=30 Oct 11 10:29:48.639038 master-1 kubenswrapper[4771]: I1011 10:29:48.638422 4771 scope.go:117] "RemoveContainer" containerID="d00f571fb5251acb052a97b0ee5169046519d14d8990c98b7ea440fa842ffd37" Oct 11 10:29:48.943194 master-1 kubenswrapper[4771]: I1011 10:29:48.942895 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:49.079931 master-1 kubenswrapper[4771]: I1011 10:29:49.079792 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-var-lock\") pod \"7662f87a-13ba-439c-b386-05e68284803c\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " Oct 11 10:29:49.079931 master-1 kubenswrapper[4771]: I1011 10:29:49.079913 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-kubelet-dir\") pod \"7662f87a-13ba-439c-b386-05e68284803c\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " Oct 11 10:29:49.080585 master-1 kubenswrapper[4771]: I1011 10:29:49.079994 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-var-lock" (OuterVolumeSpecName: "var-lock") pod "7662f87a-13ba-439c-b386-05e68284803c" (UID: "7662f87a-13ba-439c-b386-05e68284803c"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:49.080585 master-1 kubenswrapper[4771]: I1011 10:29:49.080042 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/7662f87a-13ba-439c-b386-05e68284803c-kube-api-access\") pod \"7662f87a-13ba-439c-b386-05e68284803c\" (UID: \"7662f87a-13ba-439c-b386-05e68284803c\") " Oct 11 10:29:49.080585 master-1 kubenswrapper[4771]: I1011 10:29:49.080104 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "7662f87a-13ba-439c-b386-05e68284803c" (UID: "7662f87a-13ba-439c-b386-05e68284803c"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:29:49.080585 master-1 kubenswrapper[4771]: I1011 10:29:49.080451 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:49.080585 master-1 kubenswrapper[4771]: I1011 10:29:49.080486 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7662f87a-13ba-439c-b386-05e68284803c-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:49.084531 master-1 kubenswrapper[4771]: I1011 10:29:49.084478 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7662f87a-13ba-439c-b386-05e68284803c-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "7662f87a-13ba-439c-b386-05e68284803c" (UID: "7662f87a-13ba-439c-b386-05e68284803c"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:29:49.182426 master-1 kubenswrapper[4771]: I1011 10:29:49.182223 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/7662f87a-13ba-439c-b386-05e68284803c-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:29:49.283616 master-1 kubenswrapper[4771]: I1011 10:29:49.283506 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:29:49.284200 master-1 kubenswrapper[4771]: E1011 10:29:49.283800 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:30:53.283771733 +0000 UTC m=+285.257998204 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:29:49.384600 master-1 kubenswrapper[4771]: I1011 10:29:49.384540 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:29:49.385148 master-1 kubenswrapper[4771]: E1011 10:29:49.385064 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:30:53.38486314 +0000 UTC m=+285.359089581 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:29:49.496996 master-1 kubenswrapper[4771]: I1011 10:29:49.496862 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:49.496996 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:49.496996 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:49.496996 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:49.497419 master-1 kubenswrapper[4771]: I1011 10:29:49.497384 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:49.628860 master-1 kubenswrapper[4771]: I1011 10:29:49.628751 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:29:49.629188 master-1 kubenswrapper[4771]: I1011 10:29:49.628888 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:29:49.645204 master-1 kubenswrapper[4771]: I1011 10:29:49.645148 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/0.log" Oct 11 10:29:49.648323 master-1 kubenswrapper[4771]: I1011 10:29:49.648276 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerStarted","Data":"af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde"} Oct 11 10:29:49.651299 master-1 kubenswrapper[4771]: I1011 10:29:49.651262 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-4-master-1" event={"ID":"7662f87a-13ba-439c-b386-05e68284803c","Type":"ContainerDied","Data":"ed6acc7afd35b0ef55fa3ef023c51664249170487a6297da51f4f2e72955fbfc"} Oct 11 10:29:49.651444 master-1 kubenswrapper[4771]: I1011 10:29:49.651303 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed6acc7afd35b0ef55fa3ef023c51664249170487a6297da51f4f2e72955fbfc" Oct 11 10:29:49.651444 master-1 kubenswrapper[4771]: I1011 10:29:49.651422 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-4-master-1" Oct 11 10:29:49.915701 master-1 kubenswrapper[4771]: I1011 10:29:49.914147 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65d86dff78-bg7lk"] Oct 11 10:29:49.915701 master-1 kubenswrapper[4771]: E1011 10:29:49.914394 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="7662f87a-13ba-439c-b386-05e68284803c" containerName="installer" Oct 11 10:29:49.915701 master-1 kubenswrapper[4771]: I1011 10:29:49.914407 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="7662f87a-13ba-439c-b386-05e68284803c" containerName="installer" Oct 11 10:29:49.915701 master-1 kubenswrapper[4771]: I1011 10:29:49.914495 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="7662f87a-13ba-439c-b386-05e68284803c" containerName="installer" Oct 11 10:29:49.915701 master-1 kubenswrapper[4771]: I1011 10:29:49.915169 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:49.919546 master-1 kubenswrapper[4771]: I1011 10:29:49.918984 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"metrics-server-tls" Oct 11 10:29:49.920115 master-1 kubenswrapper[4771]: I1011 10:29:49.919827 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"metrics-server-audit-profiles" Oct 11 10:29:49.920115 master-1 kubenswrapper[4771]: I1011 10:29:49.919933 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"metrics-server-ap7ej74ueigk4" Oct 11 10:29:49.920693 master-1 kubenswrapper[4771]: I1011 10:29:49.920658 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"metrics-client-certs" Oct 11 10:29:49.922224 master-1 kubenswrapper[4771]: I1011 10:29:49.922190 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"kubelet-serving-ca-bundle" Oct 11 10:29:49.926677 master-1 kubenswrapper[4771]: I1011 10:29:49.926304 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65d86dff78-bg7lk"] Oct 11 10:29:49.991201 master-1 kubenswrapper[4771]: I1011 10:29:49.991132 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-metrics-server-audit-profiles\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:49.991476 master-1 kubenswrapper[4771]: I1011 10:29:49.991307 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-client-certs\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:49.991476 master-1 kubenswrapper[4771]: I1011 10:29:49.991381 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:49.991622 master-1 kubenswrapper[4771]: I1011 10:29:49.991479 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/daf74cdb-6bdb-465a-8e3e-194e8868570f-audit-log\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:49.991622 master-1 kubenswrapper[4771]: I1011 10:29:49.991518 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-client-ca-bundle\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:49.991622 master-1 kubenswrapper[4771]: I1011 10:29:49.991549 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s2hx\" (UniqueName: \"kubernetes.io/projected/daf74cdb-6bdb-465a-8e3e-194e8868570f-kube-api-access-5s2hx\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:49.991622 master-1 kubenswrapper[4771]: I1011 10:29:49.991585 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-server-tls\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.092328 master-1 kubenswrapper[4771]: I1011 10:29:50.092253 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-client-certs\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.092328 master-1 kubenswrapper[4771]: I1011 10:29:50.092320 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.092678 master-1 kubenswrapper[4771]: I1011 10:29:50.092399 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/daf74cdb-6bdb-465a-8e3e-194e8868570f-audit-log\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.092678 master-1 kubenswrapper[4771]: I1011 10:29:50.092428 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-client-ca-bundle\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.092678 master-1 kubenswrapper[4771]: I1011 10:29:50.092451 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5s2hx\" (UniqueName: \"kubernetes.io/projected/daf74cdb-6bdb-465a-8e3e-194e8868570f-kube-api-access-5s2hx\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.092678 master-1 kubenswrapper[4771]: I1011 10:29:50.092476 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-server-tls\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.092678 master-1 kubenswrapper[4771]: I1011 10:29:50.092506 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-metrics-server-audit-profiles\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.093749 master-1 kubenswrapper[4771]: I1011 10:29:50.093589 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/daf74cdb-6bdb-465a-8e3e-194e8868570f-audit-log\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.094170 master-1 kubenswrapper[4771]: I1011 10:29:50.094081 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.095108 master-1 kubenswrapper[4771]: I1011 10:29:50.095053 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-metrics-server-audit-profiles\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.098042 master-1 kubenswrapper[4771]: I1011 10:29:50.097982 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-client-certs\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.098297 master-1 kubenswrapper[4771]: I1011 10:29:50.098245 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-client-ca-bundle\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.113729 master-1 kubenswrapper[4771]: I1011 10:29:50.113651 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-server-tls\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.128094 master-1 kubenswrapper[4771]: I1011 10:29:50.128024 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s2hx\" (UniqueName: \"kubernetes.io/projected/daf74cdb-6bdb-465a-8e3e-194e8868570f-kube-api-access-5s2hx\") pod \"metrics-server-65d86dff78-bg7lk\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.238740 master-1 kubenswrapper[4771]: I1011 10:29:50.238602 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:50.496941 master-1 kubenswrapper[4771]: I1011 10:29:50.496334 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:50.496941 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:50.496941 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:50.496941 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:50.496941 master-1 kubenswrapper[4771]: I1011 10:29:50.496915 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:50.703478 master-1 kubenswrapper[4771]: I1011 10:29:50.703419 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65d86dff78-bg7lk"] Oct 11 10:29:50.706611 master-1 kubenswrapper[4771]: W1011 10:29:50.706527 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf74cdb_6bdb_465a_8e3e_194e8868570f.slice/crio-307edc8bf8db53981b4988030525d3bc29e6569573860e0ae13cb952073e6408 WatchSource:0}: Error finding container 307edc8bf8db53981b4988030525d3bc29e6569573860e0ae13cb952073e6408: Status 404 returned error can't find the container with id 307edc8bf8db53981b4988030525d3bc29e6569573860e0ae13cb952073e6408 Oct 11 10:29:51.292239 master-1 kubenswrapper[4771]: I1011 10:29:51.292163 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1"] Oct 11 10:29:51.292816 master-1 kubenswrapper[4771]: I1011 10:29:51.292777 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:29:51.296015 master-1 kubenswrapper[4771]: I1011 10:29:51.295968 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-scheduler"/"kube-root-ca.crt" Oct 11 10:29:51.296582 master-1 kubenswrapper[4771]: I1011 10:29:51.296543 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-scheduler"/"openshift-service-ca.crt" Oct 11 10:29:51.306753 master-1 kubenswrapper[4771]: I1011 10:29:51.300531 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1"] Oct 11 10:29:51.417210 master-1 kubenswrapper[4771]: I1011 10:29:51.417100 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkcv5\" (UniqueName: \"kubernetes.io/projected/bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe-kube-api-access-vkcv5\") pod \"openshift-kube-scheduler-guard-master-1\" (UID: \"bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:29:51.496717 master-1 kubenswrapper[4771]: I1011 10:29:51.496665 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:51.496717 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:51.496717 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:51.496717 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:51.497014 master-1 kubenswrapper[4771]: I1011 10:29:51.496729 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:51.518383 master-1 kubenswrapper[4771]: I1011 10:29:51.518319 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-vkcv5\" (UniqueName: \"kubernetes.io/projected/bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe-kube-api-access-vkcv5\") pod \"openshift-kube-scheduler-guard-master-1\" (UID: \"bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:29:51.551004 master-1 kubenswrapper[4771]: I1011 10:29:51.550861 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkcv5\" (UniqueName: \"kubernetes.io/projected/bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe-kube-api-access-vkcv5\") pod \"openshift-kube-scheduler-guard-master-1\" (UID: \"bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:29:51.606268 master-1 kubenswrapper[4771]: I1011 10:29:51.606201 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:29:51.660084 master-1 kubenswrapper[4771]: I1011 10:29:51.660048 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:51.660174 master-1 kubenswrapper[4771]: I1011 10:29:51.660091 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:51.661982 master-1 kubenswrapper[4771]: I1011 10:29:51.661949 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" event={"ID":"daf74cdb-6bdb-465a-8e3e-194e8868570f","Type":"ContainerStarted","Data":"307edc8bf8db53981b4988030525d3bc29e6569573860e0ae13cb952073e6408"} Oct 11 10:29:51.694028 master-1 kubenswrapper[4771]: I1011 10:29:51.693990 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:52.500844 master-1 kubenswrapper[4771]: I1011 10:29:52.500704 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:52.500844 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:52.500844 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:52.500844 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:52.501420 master-1 kubenswrapper[4771]: I1011 10:29:52.500894 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:52.641449 master-1 kubenswrapper[4771]: I1011 10:29:52.641395 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:52.641449 master-1 kubenswrapper[4771]: I1011 10:29:52.641458 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:52.677959 master-1 kubenswrapper[4771]: I1011 10:29:52.677907 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:52.722062 master-1 kubenswrapper[4771]: I1011 10:29:52.722025 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:29:52.732775 master-1 kubenswrapper[4771]: I1011 10:29:52.732734 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:29:53.272186 master-1 kubenswrapper[4771]: I1011 10:29:53.272099 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-etcd/etcd-master-1" Oct 11 10:29:53.497551 master-1 kubenswrapper[4771]: I1011 10:29:53.497473 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:53.497551 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:53.497551 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:53.497551 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:53.497949 master-1 kubenswrapper[4771]: I1011 10:29:53.497562 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:53.840920 master-1 kubenswrapper[4771]: I1011 10:29:53.840842 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:53.840920 master-1 kubenswrapper[4771]: I1011 10:29:53.840900 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:53.876043 master-1 kubenswrapper[4771]: I1011 10:29:53.875971 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:53.978140 master-1 kubenswrapper[4771]: E1011 10:29:53.978087 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5268b2f2ae2aef0c7f2e7a6e651ed702.slice/crio-af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:29:54.452249 master-1 kubenswrapper[4771]: E1011 10:29:54.452164 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" cmd=["/bin/bash","-c","test -f /ready/ready"] Oct 11 10:29:54.454198 master-1 kubenswrapper[4771]: E1011 10:29:54.454128 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" cmd=["/bin/bash","-c","test -f /ready/ready"] Oct 11 10:29:54.455930 master-1 kubenswrapper[4771]: E1011 10:29:54.455891 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" cmd=["/bin/bash","-c","test -f /ready/ready"] Oct 11 10:29:54.456041 master-1 kubenswrapper[4771]: E1011 10:29:54.455935 4771 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" podUID="806cd59c-056a-4fb4-a3b4-cb716c01cdea" containerName="kube-multus-additional-cni-plugins" Oct 11 10:29:54.498794 master-1 kubenswrapper[4771]: I1011 10:29:54.498723 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:54.498794 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:54.498794 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:54.498794 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:54.499164 master-1 kubenswrapper[4771]: I1011 10:29:54.498844 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:54.629730 master-1 kubenswrapper[4771]: I1011 10:29:54.629635 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:29:54.629985 master-1 kubenswrapper[4771]: I1011 10:29:54.629731 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:29:54.679390 master-1 kubenswrapper[4771]: I1011 10:29:54.679307 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/1.log" Oct 11 10:29:54.682476 master-1 kubenswrapper[4771]: I1011 10:29:54.682426 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/0.log" Oct 11 10:29:54.685376 master-1 kubenswrapper[4771]: I1011 10:29:54.685286 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde" exitCode=1 Oct 11 10:29:54.685516 master-1 kubenswrapper[4771]: I1011 10:29:54.685409 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerDied","Data":"af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde"} Oct 11 10:29:54.685516 master-1 kubenswrapper[4771]: I1011 10:29:54.685499 4771 scope.go:117] "RemoveContainer" containerID="d00f571fb5251acb052a97b0ee5169046519d14d8990c98b7ea440fa842ffd37" Oct 11 10:29:54.686638 master-1 kubenswrapper[4771]: I1011 10:29:54.686584 4771 scope.go:117] "RemoveContainer" containerID="af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde" Oct 11 10:29:54.687193 master-1 kubenswrapper[4771]: E1011 10:29:54.687127 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"etcd\" with CrashLoopBackOff: \"back-off 10s restarting failed container=etcd pod=etcd-master-1_openshift-etcd(5268b2f2ae2aef0c7f2e7a6e651ed702)\"" pod="openshift-etcd/etcd-master-1" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" Oct 11 10:29:54.754406 master-1 kubenswrapper[4771]: I1011 10:29:54.754196 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59"] Oct 11 10:29:54.755818 master-1 kubenswrapper[4771]: I1011 10:29:54.755764 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:54.758295 master-1 kubenswrapper[4771]: I1011 10:29:54.758239 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:29:54.759152 master-1 kubenswrapper[4771]: I1011 10:29:54.759080 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"telemeter-client-tls" Oct 11 10:29:54.759235 master-1 kubenswrapper[4771]: I1011 10:29:54.759080 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"telemeter-client-kube-rbac-proxy-config" Oct 11 10:29:54.759389 master-1 kubenswrapper[4771]: I1011 10:29:54.759313 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"telemeter-client" Oct 11 10:29:54.761013 master-1 kubenswrapper[4771]: I1011 10:29:54.760949 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"telemeter-client-serving-certs-ca-bundle" Oct 11 10:29:54.761196 master-1 kubenswrapper[4771]: I1011 10:29:54.761023 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"federate-client-certs" Oct 11 10:29:54.765955 master-1 kubenswrapper[4771]: I1011 10:29:54.765903 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59"] Oct 11 10:29:54.769973 master-1 kubenswrapper[4771]: I1011 10:29:54.769927 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"telemeter-trusted-ca-bundle-56c9b9fa8d9gs" Oct 11 10:29:54.895205 master-1 kubenswrapper[4771]: I1011 10:29:54.895128 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1"] Oct 11 10:29:54.899626 master-1 kubenswrapper[4771]: I1011 10:29:54.899502 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hzt\" (UniqueName: \"kubernetes.io/projected/24ee422a-a8f9-436d-b2be-ee2cfa387868-kube-api-access-p6hzt\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:54.899728 master-1 kubenswrapper[4771]: I1011 10:29:54.899662 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-secret-telemeter-client\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:54.900066 master-1 kubenswrapper[4771]: I1011 10:29:54.899984 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ee422a-a8f9-436d-b2be-ee2cfa387868-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:54.900143 master-1 kubenswrapper[4771]: I1011 10:29:54.900109 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-telemeter-client-tls\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:54.900223 master-1 kubenswrapper[4771]: I1011 10:29:54.900169 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:54.900339 master-1 kubenswrapper[4771]: I1011 10:29:54.900292 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ee422a-a8f9-436d-b2be-ee2cfa387868-serving-certs-ca-bundle\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:54.900517 master-1 kubenswrapper[4771]: I1011 10:29:54.900468 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ee422a-a8f9-436d-b2be-ee2cfa387868-metrics-client-ca\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:54.900600 master-1 kubenswrapper[4771]: I1011 10:29:54.900527 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-federate-client-tls\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.001561 master-1 kubenswrapper[4771]: I1011 10:29:55.001454 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hzt\" (UniqueName: \"kubernetes.io/projected/24ee422a-a8f9-436d-b2be-ee2cfa387868-kube-api-access-p6hzt\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.001561 master-1 kubenswrapper[4771]: I1011 10:29:55.001548 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-secret-telemeter-client\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.001776 master-1 kubenswrapper[4771]: I1011 10:29:55.001600 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ee422a-a8f9-436d-b2be-ee2cfa387868-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.001776 master-1 kubenswrapper[4771]: I1011 10:29:55.001644 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.001776 master-1 kubenswrapper[4771]: I1011 10:29:55.001678 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-telemeter-client-tls\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.001776 master-1 kubenswrapper[4771]: I1011 10:29:55.001716 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ee422a-a8f9-436d-b2be-ee2cfa387868-serving-certs-ca-bundle\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.002078 master-1 kubenswrapper[4771]: I1011 10:29:55.001788 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ee422a-a8f9-436d-b2be-ee2cfa387868-metrics-client-ca\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.002078 master-1 kubenswrapper[4771]: I1011 10:29:55.001833 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-federate-client-tls\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.003438 master-1 kubenswrapper[4771]: I1011 10:29:55.003325 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ee422a-a8f9-436d-b2be-ee2cfa387868-metrics-client-ca\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.004101 master-1 kubenswrapper[4771]: I1011 10:29:55.004017 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ee422a-a8f9-436d-b2be-ee2cfa387868-serving-certs-ca-bundle\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.004546 master-1 kubenswrapper[4771]: I1011 10:29:55.004426 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ee422a-a8f9-436d-b2be-ee2cfa387868-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.007039 master-1 kubenswrapper[4771]: I1011 10:29:55.006982 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-secret-telemeter-client\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.007223 master-1 kubenswrapper[4771]: I1011 10:29:55.007157 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-telemeter-client-tls\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.007932 master-1 kubenswrapper[4771]: I1011 10:29:55.007864 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-federate-client-tls\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.008780 master-1 kubenswrapper[4771]: I1011 10:29:55.008725 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ee422a-a8f9-436d-b2be-ee2cfa387868-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.036222 master-1 kubenswrapper[4771]: I1011 10:29:55.036133 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hzt\" (UniqueName: \"kubernetes.io/projected/24ee422a-a8f9-436d-b2be-ee2cfa387868-kube-api-access-p6hzt\") pod \"telemeter-client-5b5c6cc5dd-rhh59\" (UID: \"24ee422a-a8f9-436d-b2be-ee2cfa387868\") " pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.129902 master-1 kubenswrapper[4771]: I1011 10:29:55.129814 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" Oct 11 10:29:55.454477 master-1 kubenswrapper[4771]: I1011 10:29:55.454402 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf"] Oct 11 10:29:55.455539 master-1 kubenswrapper[4771]: I1011 10:29:55.455509 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" Oct 11 10:29:55.459089 master-1 kubenswrapper[4771]: I1011 10:29:55.459030 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-multus"/"multus-admission-controller-secret" Oct 11 10:29:55.463013 master-1 kubenswrapper[4771]: I1011 10:29:55.462961 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf"] Oct 11 10:29:55.499161 master-1 kubenswrapper[4771]: I1011 10:29:55.498672 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:55.499161 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:55.499161 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:55.499161 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:55.499161 master-1 kubenswrapper[4771]: I1011 10:29:55.498778 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:55.609052 master-1 kubenswrapper[4771]: I1011 10:29:55.608951 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/5ce7321b-beff-4c96-9998-a3177ac79f36-webhook-certs\") pod \"multus-admission-controller-7b6b7bb859-rwvpf\" (UID: \"5ce7321b-beff-4c96-9998-a3177ac79f36\") " pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" Oct 11 10:29:55.609052 master-1 kubenswrapper[4771]: I1011 10:29:55.609044 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhg6\" (UniqueName: \"kubernetes.io/projected/5ce7321b-beff-4c96-9998-a3177ac79f36-kube-api-access-zvhg6\") pod \"multus-admission-controller-7b6b7bb859-rwvpf\" (UID: \"5ce7321b-beff-4c96-9998-a3177ac79f36\") " pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" Oct 11 10:29:55.710902 master-1 kubenswrapper[4771]: I1011 10:29:55.710738 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/5ce7321b-beff-4c96-9998-a3177ac79f36-webhook-certs\") pod \"multus-admission-controller-7b6b7bb859-rwvpf\" (UID: \"5ce7321b-beff-4c96-9998-a3177ac79f36\") " pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" Oct 11 10:29:55.710902 master-1 kubenswrapper[4771]: I1011 10:29:55.710821 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhg6\" (UniqueName: \"kubernetes.io/projected/5ce7321b-beff-4c96-9998-a3177ac79f36-kube-api-access-zvhg6\") pod \"multus-admission-controller-7b6b7bb859-rwvpf\" (UID: \"5ce7321b-beff-4c96-9998-a3177ac79f36\") " pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" Oct 11 10:29:55.718835 master-1 kubenswrapper[4771]: I1011 10:29:55.718159 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/5ce7321b-beff-4c96-9998-a3177ac79f36-webhook-certs\") pod \"multus-admission-controller-7b6b7bb859-rwvpf\" (UID: \"5ce7321b-beff-4c96-9998-a3177ac79f36\") " pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" Oct 11 10:29:55.734622 master-1 kubenswrapper[4771]: I1011 10:29:55.734560 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhg6\" (UniqueName: \"kubernetes.io/projected/5ce7321b-beff-4c96-9998-a3177ac79f36-kube-api-access-zvhg6\") pod \"multus-admission-controller-7b6b7bb859-rwvpf\" (UID: \"5ce7321b-beff-4c96-9998-a3177ac79f36\") " pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" Oct 11 10:29:55.775778 master-1 kubenswrapper[4771]: I1011 10:29:55.775683 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" Oct 11 10:29:56.240607 master-1 kubenswrapper[4771]: I1011 10:29:56.240531 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:29:56.241458 master-1 kubenswrapper[4771]: E1011 10:29:56.240776 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:29:56.241458 master-1 kubenswrapper[4771]: E1011 10:29:56.240924 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:31:00.240893599 +0000 UTC m=+292.215120060 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:29:56.497763 master-1 kubenswrapper[4771]: I1011 10:29:56.497571 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:56.497763 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:56.497763 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:56.497763 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:56.497763 master-1 kubenswrapper[4771]: I1011 10:29:56.497670 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:56.719730 master-1 kubenswrapper[4771]: I1011 10:29:56.718759 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" event={"ID":"daf74cdb-6bdb-465a-8e3e-194e8868570f","Type":"ContainerStarted","Data":"d71774e5747fba198d1f1c685867c43372766be8110c50262b34cb5aee247b7d"} Oct 11 10:29:56.719730 master-1 kubenswrapper[4771]: I1011 10:29:56.718914 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:29:56.720746 master-1 kubenswrapper[4771]: I1011 10:29:56.720705 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/1.log" Oct 11 10:29:56.955652 master-1 kubenswrapper[4771]: I1011 10:29:56.955531 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" podStartSLOduration=2.106971813 podStartE2EDuration="7.955495798s" podCreationTimestamp="2025-10-11 10:29:49 +0000 UTC" firstStartedPulling="2025-10-11 10:29:50.70963954 +0000 UTC m=+222.683866011" lastFinishedPulling="2025-10-11 10:29:56.558163535 +0000 UTC m=+228.532389996" observedRunningTime="2025-10-11 10:29:56.737878579 +0000 UTC m=+228.712105030" watchObservedRunningTime="2025-10-11 10:29:56.955495798 +0000 UTC m=+228.929722269" Oct 11 10:29:56.959735 master-1 kubenswrapper[4771]: I1011 10:29:56.959694 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1"] Oct 11 10:29:57.012719 master-1 kubenswrapper[4771]: I1011 10:29:57.012654 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf"] Oct 11 10:29:57.016377 master-1 kubenswrapper[4771]: I1011 10:29:57.015692 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59"] Oct 11 10:29:57.032240 master-1 kubenswrapper[4771]: W1011 10:29:57.032198 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ee422a_a8f9_436d_b2be_ee2cfa387868.slice/crio-a4590572e64d77528c45fc16250bd4f7576fcf527774c6298823721b8c5268fd WatchSource:0}: Error finding container a4590572e64d77528c45fc16250bd4f7576fcf527774c6298823721b8c5268fd: Status 404 returned error can't find the container with id a4590572e64d77528c45fc16250bd4f7576fcf527774c6298823721b8c5268fd Oct 11 10:29:57.499843 master-1 kubenswrapper[4771]: I1011 10:29:57.499771 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:57.499843 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:57.499843 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:57.499843 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:57.501095 master-1 kubenswrapper[4771]: I1011 10:29:57.499873 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:57.732337 master-1 kubenswrapper[4771]: I1011 10:29:57.732240 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" event={"ID":"5ce7321b-beff-4c96-9998-a3177ac79f36","Type":"ContainerStarted","Data":"965a15ffe7b54d5e8a5dee16cc0508a1fdef2cf049f7f7b216482accd5552ba0"} Oct 11 10:29:57.734308 master-1 kubenswrapper[4771]: I1011 10:29:57.734251 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" event={"ID":"bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe","Type":"ContainerStarted","Data":"f0fea7d6a8eafab0f8489dde0fdb5801048d55c50b2e9df820711886dd128bb4"} Oct 11 10:29:57.736267 master-1 kubenswrapper[4771]: I1011 10:29:57.736166 4771 generic.go:334] "Generic (PLEG): container finished" podID="89fad8183e18ab3ad0c46d272335e5f8" containerID="2a73de07f276bd8a0b93475494fdae31f01c7c950b265a424f35d3d72462410c" exitCode=0 Oct 11 10:29:57.736351 master-1 kubenswrapper[4771]: I1011 10:29:57.736291 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"89fad8183e18ab3ad0c46d272335e5f8","Type":"ContainerDied","Data":"2a73de07f276bd8a0b93475494fdae31f01c7c950b265a424f35d3d72462410c"} Oct 11 10:29:57.738755 master-1 kubenswrapper[4771]: I1011 10:29:57.738703 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" event={"ID":"24ee422a-a8f9-436d-b2be-ee2cfa387868","Type":"ContainerStarted","Data":"a4590572e64d77528c45fc16250bd4f7576fcf527774c6298823721b8c5268fd"} Oct 11 10:29:58.273088 master-1 kubenswrapper[4771]: I1011 10:29:58.273019 4771 kubelet.go:2542] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-etcd/etcd-master-1" Oct 11 10:29:58.273247 master-1 kubenswrapper[4771]: I1011 10:29:58.273112 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-etcd/etcd-master-1" Oct 11 10:29:58.273247 master-1 kubenswrapper[4771]: I1011 10:29:58.273137 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-etcd/etcd-master-1" Oct 11 10:29:58.274204 master-1 kubenswrapper[4771]: I1011 10:29:58.274161 4771 scope.go:117] "RemoveContainer" containerID="af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde" Oct 11 10:29:58.274672 master-1 kubenswrapper[4771]: E1011 10:29:58.274628 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"etcd\" with CrashLoopBackOff: \"back-off 10s restarting failed container=etcd pod=etcd-master-1_openshift-etcd(5268b2f2ae2aef0c7f2e7a6e651ed702)\"" pod="openshift-etcd/etcd-master-1" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" Oct 11 10:29:58.498988 master-1 kubenswrapper[4771]: I1011 10:29:58.498892 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:58.498988 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:58.498988 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:58.498988 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:58.499309 master-1 kubenswrapper[4771]: I1011 10:29:58.499033 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:58.668918 master-1 kubenswrapper[4771]: I1011 10:29:58.668857 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:29:58.669449 master-1 kubenswrapper[4771]: E1011 10:29:58.669009 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:29:58.669449 master-1 kubenswrapper[4771]: E1011 10:29:58.669085 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:31:02.669066365 +0000 UTC m=+294.643292806 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:29:59.497435 master-1 kubenswrapper[4771]: I1011 10:29:59.497314 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:29:59.497435 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:29:59.497435 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:29:59.497435 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:29:59.497435 master-1 kubenswrapper[4771]: I1011 10:29:59.497430 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:29:59.630350 master-1 kubenswrapper[4771]: I1011 10:29:59.630234 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:29:59.630677 master-1 kubenswrapper[4771]: I1011 10:29:59.630386 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:29:59.755019 master-1 kubenswrapper[4771]: I1011 10:29:59.754874 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" event={"ID":"bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe","Type":"ContainerStarted","Data":"bb2efb4d50d7aa4eebda9d5e309a23e278332e08f60110025acc022441421550"} Oct 11 10:30:00.167268 master-1 kubenswrapper[4771]: I1011 10:30:00.167180 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v"] Oct 11 10:30:00.168145 master-1 kubenswrapper[4771]: I1011 10:30:00.168090 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.171893 master-1 kubenswrapper[4771]: I1011 10:30:00.171833 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-operator-lifecycle-manager"/"collect-profiles-config" Oct 11 10:30:00.171893 master-1 kubenswrapper[4771]: I1011 10:30:00.171890 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-operator-lifecycle-manager"/"pprof-cert" Oct 11 10:30:00.178027 master-1 kubenswrapper[4771]: I1011 10:30:00.177922 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v"] Oct 11 10:30:00.191441 master-1 kubenswrapper[4771]: I1011 10:30:00.191378 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf56n\" (UniqueName: \"kubernetes.io/projected/f4904d67-3c44-40d9-8ea8-026d727e9486-kube-api-access-cf56n\") pod \"collect-profiles-29336310-8nc4v\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.191799 master-1 kubenswrapper[4771]: I1011 10:30:00.191725 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/f4904d67-3c44-40d9-8ea8-026d727e9486-secret-volume\") pod \"collect-profiles-29336310-8nc4v\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.191966 master-1 kubenswrapper[4771]: I1011 10:30:00.191915 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4904d67-3c44-40d9-8ea8-026d727e9486-config-volume\") pod \"collect-profiles-29336310-8nc4v\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.293210 master-1 kubenswrapper[4771]: I1011 10:30:00.293089 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-cf56n\" (UniqueName: \"kubernetes.io/projected/f4904d67-3c44-40d9-8ea8-026d727e9486-kube-api-access-cf56n\") pod \"collect-profiles-29336310-8nc4v\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.293556 master-1 kubenswrapper[4771]: I1011 10:30:00.293258 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/f4904d67-3c44-40d9-8ea8-026d727e9486-secret-volume\") pod \"collect-profiles-29336310-8nc4v\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.293556 master-1 kubenswrapper[4771]: I1011 10:30:00.293329 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4904d67-3c44-40d9-8ea8-026d727e9486-config-volume\") pod \"collect-profiles-29336310-8nc4v\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.295022 master-1 kubenswrapper[4771]: I1011 10:30:00.294958 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4904d67-3c44-40d9-8ea8-026d727e9486-config-volume\") pod \"collect-profiles-29336310-8nc4v\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.298895 master-1 kubenswrapper[4771]: I1011 10:30:00.298831 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/f4904d67-3c44-40d9-8ea8-026d727e9486-secret-volume\") pod \"collect-profiles-29336310-8nc4v\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.324337 master-1 kubenswrapper[4771]: I1011 10:30:00.324242 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf56n\" (UniqueName: \"kubernetes.io/projected/f4904d67-3c44-40d9-8ea8-026d727e9486-kube-api-access-cf56n\") pod \"collect-profiles-29336310-8nc4v\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.491340 master-1 kubenswrapper[4771]: I1011 10:30:00.491151 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:00.510781 master-1 kubenswrapper[4771]: I1011 10:30:00.510689 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:00.510781 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:00.510781 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:00.510781 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:00.511149 master-1 kubenswrapper[4771]: I1011 10:30:00.510778 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:00.778437 master-1 kubenswrapper[4771]: I1011 10:30:00.778234 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"89fad8183e18ab3ad0c46d272335e5f8","Type":"ContainerStarted","Data":"0400db595d18039edaf6ab7ccb3c1b1a3510ae9588fc33a6a91a15e993a6d1a4"} Oct 11 10:30:00.779207 master-1 kubenswrapper[4771]: I1011 10:30:00.778529 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:30:00.785156 master-1 kubenswrapper[4771]: I1011 10:30:00.785103 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:30:00.797546 master-1 kubenswrapper[4771]: I1011 10:30:00.797435 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podStartSLOduration=9.797402467 podStartE2EDuration="9.797402467s" podCreationTimestamp="2025-10-11 10:29:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:30:00.795023551 +0000 UTC m=+232.769250032" watchObservedRunningTime="2025-10-11 10:30:00.797402467 +0000 UTC m=+232.771628938" Oct 11 10:30:00.939170 master-1 kubenswrapper[4771]: I1011 10:30:00.939092 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v"] Oct 11 10:30:00.946657 master-1 kubenswrapper[4771]: W1011 10:30:00.946582 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4904d67_3c44_40d9_8ea8_026d727e9486.slice/crio-fb9abe4abde9e0f6d44c16dbc1ef22f8a3f53fff1d42a62d4fb8563051892a8d WatchSource:0}: Error finding container fb9abe4abde9e0f6d44c16dbc1ef22f8a3f53fff1d42a62d4fb8563051892a8d: Status 404 returned error can't find the container with id fb9abe4abde9e0f6d44c16dbc1ef22f8a3f53fff1d42a62d4fb8563051892a8d Oct 11 10:30:01.496869 master-1 kubenswrapper[4771]: I1011 10:30:01.496783 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:01.496869 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:01.496869 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:01.496869 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:01.497223 master-1 kubenswrapper[4771]: I1011 10:30:01.496880 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:01.785228 master-1 kubenswrapper[4771]: I1011 10:30:01.785162 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" event={"ID":"f4904d67-3c44-40d9-8ea8-026d727e9486","Type":"ContainerStarted","Data":"fb9abe4abde9e0f6d44c16dbc1ef22f8a3f53fff1d42a62d4fb8563051892a8d"} Oct 11 10:30:02.496073 master-1 kubenswrapper[4771]: I1011 10:30:02.495944 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:02.496073 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:02.496073 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:02.496073 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:02.496073 master-1 kubenswrapper[4771]: I1011 10:30:02.496026 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:02.794711 master-1 kubenswrapper[4771]: I1011 10:30:02.794514 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" event={"ID":"f4904d67-3c44-40d9-8ea8-026d727e9486","Type":"ContainerStarted","Data":"ea50bb78d4de53e43e9be3f2830fede428957c124838ed0305c9a99b641c0252"} Oct 11 10:30:02.798381 master-1 kubenswrapper[4771]: I1011 10:30:02.798251 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"89fad8183e18ab3ad0c46d272335e5f8","Type":"ContainerStarted","Data":"27a52449e5ec1bd52177b8ae4e5229c8bc4e5a7be149b07a0e7cb307be3932da"} Oct 11 10:30:03.498124 master-1 kubenswrapper[4771]: I1011 10:30:03.498047 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:03.498124 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:03.498124 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:03.498124 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:03.498570 master-1 kubenswrapper[4771]: I1011 10:30:03.498141 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:03.822949 master-1 kubenswrapper[4771]: I1011 10:30:03.822790 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" podStartSLOduration=3.822765536 podStartE2EDuration="3.822765536s" podCreationTimestamp="2025-10-11 10:30:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:30:03.821218814 +0000 UTC m=+235.795445345" watchObservedRunningTime="2025-10-11 10:30:03.822765536 +0000 UTC m=+235.796992017" Oct 11 10:30:04.452594 master-1 kubenswrapper[4771]: E1011 10:30:04.452443 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" cmd=["/bin/bash","-c","test -f /ready/ready"] Oct 11 10:30:04.454777 master-1 kubenswrapper[4771]: E1011 10:30:04.454739 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" cmd=["/bin/bash","-c","test -f /ready/ready"] Oct 11 10:30:04.456272 master-1 kubenswrapper[4771]: E1011 10:30:04.456230 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" cmd=["/bin/bash","-c","test -f /ready/ready"] Oct 11 10:30:04.456331 master-1 kubenswrapper[4771]: E1011 10:30:04.456273 4771 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" podUID="806cd59c-056a-4fb4-a3b4-cb716c01cdea" containerName="kube-multus-additional-cni-plugins" Oct 11 10:30:04.499378 master-1 kubenswrapper[4771]: I1011 10:30:04.499299 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:04.499378 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:04.499378 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:04.499378 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:04.499706 master-1 kubenswrapper[4771]: I1011 10:30:04.499424 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:04.631181 master-1 kubenswrapper[4771]: I1011 10:30:04.630604 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:30:04.631181 master-1 kubenswrapper[4771]: I1011 10:30:04.630693 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:30:04.809374 master-1 kubenswrapper[4771]: I1011 10:30:04.809277 4771 generic.go:334] "Generic (PLEG): container finished" podID="f4904d67-3c44-40d9-8ea8-026d727e9486" containerID="ea50bb78d4de53e43e9be3f2830fede428957c124838ed0305c9a99b641c0252" exitCode=0 Oct 11 10:30:04.809618 master-1 kubenswrapper[4771]: I1011 10:30:04.809376 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" event={"ID":"f4904d67-3c44-40d9-8ea8-026d727e9486","Type":"ContainerDied","Data":"ea50bb78d4de53e43e9be3f2830fede428957c124838ed0305c9a99b641c0252"} Oct 11 10:30:05.497125 master-1 kubenswrapper[4771]: I1011 10:30:05.496944 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:05.497125 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:05.497125 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:05.497125 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:05.497125 master-1 kubenswrapper[4771]: I1011 10:30:05.497064 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:05.818949 master-1 kubenswrapper[4771]: I1011 10:30:05.818871 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"89fad8183e18ab3ad0c46d272335e5f8","Type":"ContainerStarted","Data":"4f12c3536caf37d890a386fecb2c94e5fc57775602e9a539771326b213c3ae7e"} Oct 11 10:30:05.837849 master-1 kubenswrapper[4771]: I1011 10:30:05.837758 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podStartSLOduration=19.837729302 podStartE2EDuration="19.837729302s" podCreationTimestamp="2025-10-11 10:29:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:30:05.83655462 +0000 UTC m=+237.810781131" watchObservedRunningTime="2025-10-11 10:30:05.837729302 +0000 UTC m=+237.811955753" Oct 11 10:30:06.497066 master-1 kubenswrapper[4771]: I1011 10:30:06.496937 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:06.497066 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:06.497066 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:06.497066 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:06.497066 master-1 kubenswrapper[4771]: I1011 10:30:06.497041 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:06.678843 master-1 kubenswrapper[4771]: I1011 10:30:06.678798 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:06.690002 master-1 kubenswrapper[4771]: I1011 10:30:06.689956 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4904d67-3c44-40d9-8ea8-026d727e9486-config-volume\") pod \"f4904d67-3c44-40d9-8ea8-026d727e9486\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " Oct 11 10:30:06.690166 master-1 kubenswrapper[4771]: I1011 10:30:06.690045 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/f4904d67-3c44-40d9-8ea8-026d727e9486-secret-volume\") pod \"f4904d67-3c44-40d9-8ea8-026d727e9486\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " Oct 11 10:30:06.690207 master-1 kubenswrapper[4771]: I1011 10:30:06.690190 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf56n\" (UniqueName: \"kubernetes.io/projected/f4904d67-3c44-40d9-8ea8-026d727e9486-kube-api-access-cf56n\") pod \"f4904d67-3c44-40d9-8ea8-026d727e9486\" (UID: \"f4904d67-3c44-40d9-8ea8-026d727e9486\") " Oct 11 10:30:06.690653 master-1 kubenswrapper[4771]: I1011 10:30:06.690609 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4904d67-3c44-40d9-8ea8-026d727e9486-config-volume" (OuterVolumeSpecName: "config-volume") pod "f4904d67-3c44-40d9-8ea8-026d727e9486" (UID: "f4904d67-3c44-40d9-8ea8-026d727e9486"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:30:06.693568 master-1 kubenswrapper[4771]: I1011 10:30:06.693511 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4904d67-3c44-40d9-8ea8-026d727e9486-secret-volume" (OuterVolumeSpecName: "secret-volume") pod "f4904d67-3c44-40d9-8ea8-026d727e9486" (UID: "f4904d67-3c44-40d9-8ea8-026d727e9486"). InnerVolumeSpecName "secret-volume". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:30:06.695405 master-1 kubenswrapper[4771]: I1011 10:30:06.694598 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4904d67-3c44-40d9-8ea8-026d727e9486-kube-api-access-cf56n" (OuterVolumeSpecName: "kube-api-access-cf56n") pod "f4904d67-3c44-40d9-8ea8-026d727e9486" (UID: "f4904d67-3c44-40d9-8ea8-026d727e9486"). InnerVolumeSpecName "kube-api-access-cf56n". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:30:06.792700 master-1 kubenswrapper[4771]: I1011 10:30:06.792557 4771 reconciler_common.go:293] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4904d67-3c44-40d9-8ea8-026d727e9486-config-volume\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:06.792700 master-1 kubenswrapper[4771]: I1011 10:30:06.792621 4771 reconciler_common.go:293] "Volume detached for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/f4904d67-3c44-40d9-8ea8-026d727e9486-secret-volume\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:06.792700 master-1 kubenswrapper[4771]: I1011 10:30:06.792645 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-cf56n\" (UniqueName: \"kubernetes.io/projected/f4904d67-3c44-40d9-8ea8-026d727e9486-kube-api-access-cf56n\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:06.826783 master-1 kubenswrapper[4771]: I1011 10:30:06.826570 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" event={"ID":"f4904d67-3c44-40d9-8ea8-026d727e9486","Type":"ContainerDied","Data":"fb9abe4abde9e0f6d44c16dbc1ef22f8a3f53fff1d42a62d4fb8563051892a8d"} Oct 11 10:30:06.826783 master-1 kubenswrapper[4771]: I1011 10:30:06.826624 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v" Oct 11 10:30:06.826783 master-1 kubenswrapper[4771]: I1011 10:30:06.826637 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb9abe4abde9e0f6d44c16dbc1ef22f8a3f53fff1d42a62d4fb8563051892a8d" Oct 11 10:30:06.826783 master-1 kubenswrapper[4771]: I1011 10:30:06.826759 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:30:07.497808 master-1 kubenswrapper[4771]: I1011 10:30:07.497694 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:07.497808 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:07.497808 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:07.497808 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:07.497808 master-1 kubenswrapper[4771]: I1011 10:30:07.497793 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:08.496800 master-1 kubenswrapper[4771]: I1011 10:30:08.496725 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:08.496800 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:08.496800 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:08.496800 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:08.497248 master-1 kubenswrapper[4771]: I1011 10:30:08.496818 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:09.438037 master-1 kubenswrapper[4771]: I1011 10:30:09.437852 4771 scope.go:117] "RemoveContainer" containerID="af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde" Oct 11 10:30:09.496908 master-1 kubenswrapper[4771]: I1011 10:30:09.496843 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:09.496908 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:09.496908 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:09.496908 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:09.497273 master-1 kubenswrapper[4771]: I1011 10:30:09.496912 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:09.631414 master-1 kubenswrapper[4771]: I1011 10:30:09.631313 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:30:09.631414 master-1 kubenswrapper[4771]: I1011 10:30:09.631395 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:30:10.497071 master-1 kubenswrapper[4771]: I1011 10:30:10.496981 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:10.497071 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:10.497071 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:10.497071 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:10.498186 master-1 kubenswrapper[4771]: I1011 10:30:10.497091 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:11.496858 master-1 kubenswrapper[4771]: I1011 10:30:11.496789 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:11.496858 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:11.496858 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:11.496858 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:11.497499 master-1 kubenswrapper[4771]: I1011 10:30:11.496869 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:12.496885 master-1 kubenswrapper[4771]: I1011 10:30:12.496810 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:12.496885 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:12.496885 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:12.496885 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:12.498195 master-1 kubenswrapper[4771]: I1011 10:30:12.496903 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:12.864265 master-1 kubenswrapper[4771]: I1011 10:30:12.864167 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/1.log" Oct 11 10:30:12.867677 master-1 kubenswrapper[4771]: I1011 10:30:12.867588 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"5268b2f2ae2aef0c7f2e7a6e651ed702","Type":"ContainerStarted","Data":"8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e"} Oct 11 10:30:12.870995 master-1 kubenswrapper[4771]: I1011 10:30:12.870920 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" event={"ID":"5ce7321b-beff-4c96-9998-a3177ac79f36","Type":"ContainerStarted","Data":"e3778718c55abd380f89d429871aa3167dd83cf5f32ed7e3ae6c0059601b60c2"} Oct 11 10:30:12.871147 master-1 kubenswrapper[4771]: I1011 10:30:12.870995 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" event={"ID":"5ce7321b-beff-4c96-9998-a3177ac79f36","Type":"ContainerStarted","Data":"76a63630e5dd4a315944c4777a18d2b03bde842d5b787f9b071acb9666f6fe9e"} Oct 11 10:30:12.873320 master-1 kubenswrapper[4771]: I1011 10:30:12.873256 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" event={"ID":"24ee422a-a8f9-436d-b2be-ee2cfa387868","Type":"ContainerStarted","Data":"8f225daf7060a64c85df526b1cea554f81def1c7cf4a4700bb4d653dc8571f96"} Oct 11 10:30:12.928234 master-1 kubenswrapper[4771]: I1011 10:30:12.928098 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-admission-controller-7b6b7bb859-rwvpf" podStartSLOduration=3.141481491 podStartE2EDuration="17.92806799s" podCreationTimestamp="2025-10-11 10:29:55 +0000 UTC" firstStartedPulling="2025-10-11 10:29:57.027127473 +0000 UTC m=+229.001353914" lastFinishedPulling="2025-10-11 10:30:11.813713982 +0000 UTC m=+243.787940413" observedRunningTime="2025-10-11 10:30:12.925012076 +0000 UTC m=+244.899238587" watchObservedRunningTime="2025-10-11 10:30:12.92806799 +0000 UTC m=+244.902294461" Oct 11 10:30:13.273061 master-1 kubenswrapper[4771]: I1011 10:30:13.272862 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-etcd/etcd-master-1" Oct 11 10:30:13.497676 master-1 kubenswrapper[4771]: I1011 10:30:13.497601 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:13.497676 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:13.497676 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:13.497676 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:13.498828 master-1 kubenswrapper[4771]: I1011 10:30:13.497692 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:13.699581 master-1 kubenswrapper[4771]: I1011 10:30:13.699499 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5ddb89f76-z5t6x_04cd4a19-2532-43d1-9144-1f59d9e52d19/router/0.log" Oct 11 10:30:14.295141 master-1 kubenswrapper[4771]: I1011 10:30:14.295071 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-oauth-apiserver_apiserver-65b6f4d4c9-skwvw_004ee387-d0e9-4582-ad14-f571832ebd6e/fix-audit-permissions/0.log" Oct 11 10:30:14.451817 master-1 kubenswrapper[4771]: E1011 10:30:14.451708 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" cmd=["/bin/bash","-c","test -f /ready/ready"] Oct 11 10:30:14.454005 master-1 kubenswrapper[4771]: E1011 10:30:14.453901 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" cmd=["/bin/bash","-c","test -f /ready/ready"] Oct 11 10:30:14.456599 master-1 kubenswrapper[4771]: E1011 10:30:14.456501 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" cmd=["/bin/bash","-c","test -f /ready/ready"] Oct 11 10:30:14.456748 master-1 kubenswrapper[4771]: E1011 10:30:14.456606 4771 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" podUID="806cd59c-056a-4fb4-a3b4-cb716c01cdea" containerName="kube-multus-additional-cni-plugins" Oct 11 10:30:14.496915 master-1 kubenswrapper[4771]: I1011 10:30:14.496687 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:14.496915 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:14.496915 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:14.496915 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:14.496915 master-1 kubenswrapper[4771]: I1011 10:30:14.496765 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:14.502601 master-1 kubenswrapper[4771]: I1011 10:30:14.502396 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-oauth-apiserver_apiserver-65b6f4d4c9-skwvw_004ee387-d0e9-4582-ad14-f571832ebd6e/oauth-apiserver/0.log" Oct 11 10:30:14.631852 master-1 kubenswrapper[4771]: I1011 10:30:14.631643 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:30:14.631852 master-1 kubenswrapper[4771]: I1011 10:30:14.631769 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:30:14.889388 master-1 kubenswrapper[4771]: I1011 10:30:14.889241 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" event={"ID":"24ee422a-a8f9-436d-b2be-ee2cfa387868","Type":"ContainerStarted","Data":"045908b7671941d85cb856e82c18281ded8c47a82fa48745d298d16204847f5f"} Oct 11 10:30:14.889388 master-1 kubenswrapper[4771]: I1011 10:30:14.889342 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" event={"ID":"24ee422a-a8f9-436d-b2be-ee2cfa387868","Type":"ContainerStarted","Data":"610e3fd9a365564eea8cab0976a54e60fd08654c507389a3dcc5428ff493223c"} Oct 11 10:30:14.923916 master-1 kubenswrapper[4771]: I1011 10:30:14.923742 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b5c6cc5dd-rhh59" podStartSLOduration=3.990968253 podStartE2EDuration="20.923705505s" podCreationTimestamp="2025-10-11 10:29:54 +0000 UTC" firstStartedPulling="2025-10-11 10:29:57.03465222 +0000 UTC m=+229.008878671" lastFinishedPulling="2025-10-11 10:30:13.967389472 +0000 UTC m=+245.941615923" observedRunningTime="2025-10-11 10:30:14.918211853 +0000 UTC m=+246.892438384" watchObservedRunningTime="2025-10-11 10:30:14.923705505 +0000 UTC m=+246.897931986" Oct 11 10:30:15.093817 master-1 kubenswrapper[4771]: I1011 10:30:15.093716 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-guard-master-1_3fc4970d-4f34-4fc6-9791-6218f8e42eb9/guard/0.log" Oct 11 10:30:15.498045 master-1 kubenswrapper[4771]: I1011 10:30:15.497940 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:15.498045 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:15.498045 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:15.498045 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:15.498045 master-1 kubenswrapper[4771]: I1011 10:30:15.498037 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:16.095680 master-1 kubenswrapper[4771]: I1011 10:30:16.095623 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/1.log" Oct 11 10:30:16.497255 master-1 kubenswrapper[4771]: I1011 10:30:16.497108 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:16.497255 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:16.497255 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:16.497255 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:16.497255 master-1 kubenswrapper[4771]: I1011 10:30:16.497222 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:16.896887 master-1 kubenswrapper[4771]: I1011 10:30:16.896805 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/setup/0.log" Oct 11 10:30:16.903474 master-1 kubenswrapper[4771]: I1011 10:30:16.903414 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-3-master-1_776c3745-2f4c-4a78-b1cd-77a7a1532df3/installer/0.log" Oct 11 10:30:16.903565 master-1 kubenswrapper[4771]: I1011 10:30:16.903521 4771 generic.go:334] "Generic (PLEG): container finished" podID="776c3745-2f4c-4a78-b1cd-77a7a1532df3" containerID="2a2c47f6b163a67c15dfe1ca6c1ec25571de95f1ae3f653d4b9ded6b99ad45a9" exitCode=1 Oct 11 10:30:16.903614 master-1 kubenswrapper[4771]: I1011 10:30:16.903590 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-3-master-1" event={"ID":"776c3745-2f4c-4a78-b1cd-77a7a1532df3","Type":"ContainerDied","Data":"2a2c47f6b163a67c15dfe1ca6c1ec25571de95f1ae3f653d4b9ded6b99ad45a9"} Oct 11 10:30:17.095955 master-1 kubenswrapper[4771]: I1011 10:30:17.095906 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-ensure-env-vars/0.log" Oct 11 10:30:17.296394 master-1 kubenswrapper[4771]: I1011 10:30:17.296280 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-resources-copy/0.log" Oct 11 10:30:17.494973 master-1 kubenswrapper[4771]: I1011 10:30:17.494875 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcdctl/0.log" Oct 11 10:30:17.496736 master-1 kubenswrapper[4771]: I1011 10:30:17.496680 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:17.496736 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:17.496736 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:17.496736 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:17.496881 master-1 kubenswrapper[4771]: I1011 10:30:17.496764 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:17.703562 master-1 kubenswrapper[4771]: I1011 10:30:17.703474 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/2.log" Oct 11 10:30:17.797971 master-1 kubenswrapper[4771]: I1011 10:30:17.797619 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:30:17.901732 master-1 kubenswrapper[4771]: I1011 10:30:17.901675 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-metrics/0.log" Oct 11 10:30:18.102269 master-1 kubenswrapper[4771]: I1011 10:30:18.102218 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-readyz/0.log" Oct 11 10:30:18.232211 master-1 kubenswrapper[4771]: I1011 10:30:18.232130 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-3-master-1_776c3745-2f4c-4a78-b1cd-77a7a1532df3/installer/0.log" Oct 11 10:30:18.232211 master-1 kubenswrapper[4771]: I1011 10:30:18.232205 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:30:18.272114 master-1 kubenswrapper[4771]: I1011 10:30:18.271994 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-etcd/etcd-master-1" Oct 11 10:30:18.295392 master-1 kubenswrapper[4771]: I1011 10:30:18.295320 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-rev/0.log" Oct 11 10:30:18.340390 master-1 kubenswrapper[4771]: I1011 10:30:18.340318 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kube-api-access\") pod \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " Oct 11 10:30:18.340570 master-1 kubenswrapper[4771]: I1011 10:30:18.340432 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-var-lock\") pod \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " Oct 11 10:30:18.340570 master-1 kubenswrapper[4771]: I1011 10:30:18.340498 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-var-lock" (OuterVolumeSpecName: "var-lock") pod "776c3745-2f4c-4a78-b1cd-77a7a1532df3" (UID: "776c3745-2f4c-4a78-b1cd-77a7a1532df3"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:30:18.340934 master-1 kubenswrapper[4771]: I1011 10:30:18.340866 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kubelet-dir\") pod \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\" (UID: \"776c3745-2f4c-4a78-b1cd-77a7a1532df3\") " Oct 11 10:30:18.341165 master-1 kubenswrapper[4771]: I1011 10:30:18.341065 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "776c3745-2f4c-4a78-b1cd-77a7a1532df3" (UID: "776c3745-2f4c-4a78-b1cd-77a7a1532df3"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:30:18.341799 master-1 kubenswrapper[4771]: I1011 10:30:18.341749 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:18.341855 master-1 kubenswrapper[4771]: I1011 10:30:18.341810 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:18.345117 master-1 kubenswrapper[4771]: I1011 10:30:18.345064 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "776c3745-2f4c-4a78-b1cd-77a7a1532df3" (UID: "776c3745-2f4c-4a78-b1cd-77a7a1532df3"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:30:18.442715 master-1 kubenswrapper[4771]: I1011 10:30:18.442633 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/776c3745-2f4c-4a78-b1cd-77a7a1532df3-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:18.497492 master-1 kubenswrapper[4771]: I1011 10:30:18.497399 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:18.497492 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:18.497492 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:18.497492 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:18.497492 master-1 kubenswrapper[4771]: I1011 10:30:18.497475 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:18.503609 master-1 kubenswrapper[4771]: I1011 10:30:18.503549 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_installer-1-master-1_826e1279-bc0d-426e-b6e0-5108268f340e/installer/0.log" Oct 11 10:30:18.675067 master-1 kubenswrapper[4771]: I1011 10:30:18.674996 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:30:18.738333 master-1 kubenswrapper[4771]: E1011 10:30:18.738228 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806cd59c_056a_4fb4_a3b4_cb716c01cdea.slice/crio-5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod776c3745_2f4c_4a78_b1cd_77a7a1532df3.slice/crio-conmon-2a2c47f6b163a67c15dfe1ca6c1ec25571de95f1ae3f653d4b9ded6b99ad45a9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod776c3745_2f4c_4a78_b1cd_77a7a1532df3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod776c3745_2f4c_4a78_b1cd_77a7a1532df3.slice/crio-2a2c47f6b163a67c15dfe1ca6c1ec25571de95f1ae3f653d4b9ded6b99ad45a9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod776c3745_2f4c_4a78_b1cd_77a7a1532df3.slice/crio-ccc3bd0dec107a1860fda6d334afbd26f993ae533a829286bea5468aeefd8bf7\": RecentStats: unable to find data in memory cache]" Oct 11 10:30:18.738656 master-1 kubenswrapper[4771]: E1011 10:30:18.738479 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806cd59c_056a_4fb4_a3b4_cb716c01cdea.slice/crio-conmon-5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:30:18.777150 master-1 kubenswrapper[4771]: I1011 10:30:18.776993 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-9d7j4_806cd59c-056a-4fb4-a3b4-cb716c01cdea/kube-multus-additional-cni-plugins/0.log" Oct 11 10:30:18.777150 master-1 kubenswrapper[4771]: I1011 10:30:18.777066 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:30:18.917098 master-1 kubenswrapper[4771]: I1011 10:30:18.916914 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-9d7j4_806cd59c-056a-4fb4-a3b4-cb716c01cdea/kube-multus-additional-cni-plugins/0.log" Oct 11 10:30:18.917098 master-1 kubenswrapper[4771]: I1011 10:30:18.917013 4771 generic.go:334] "Generic (PLEG): container finished" podID="806cd59c-056a-4fb4-a3b4-cb716c01cdea" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" exitCode=137 Oct 11 10:30:18.917563 master-1 kubenswrapper[4771]: I1011 10:30:18.917132 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" Oct 11 10:30:18.917563 master-1 kubenswrapper[4771]: I1011 10:30:18.917156 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" event={"ID":"806cd59c-056a-4fb4-a3b4-cb716c01cdea","Type":"ContainerDied","Data":"5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32"} Oct 11 10:30:18.917563 master-1 kubenswrapper[4771]: I1011 10:30:18.917243 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-9d7j4" event={"ID":"806cd59c-056a-4fb4-a3b4-cb716c01cdea","Type":"ContainerDied","Data":"7147f14021bd8181c058d8c3ce2203cdae664d32eab5196f21ee167281d79073"} Oct 11 10:30:18.917563 master-1 kubenswrapper[4771]: I1011 10:30:18.917278 4771 scope.go:117] "RemoveContainer" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" Oct 11 10:30:18.920226 master-1 kubenswrapper[4771]: I1011 10:30:18.920047 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-3-master-1_776c3745-2f4c-4a78-b1cd-77a7a1532df3/installer/0.log" Oct 11 10:30:18.920226 master-1 kubenswrapper[4771]: I1011 10:30:18.920117 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-3-master-1" event={"ID":"776c3745-2f4c-4a78-b1cd-77a7a1532df3","Type":"ContainerDied","Data":"ccc3bd0dec107a1860fda6d334afbd26f993ae533a829286bea5468aeefd8bf7"} Oct 11 10:30:18.920226 master-1 kubenswrapper[4771]: I1011 10:30:18.920148 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccc3bd0dec107a1860fda6d334afbd26f993ae533a829286bea5468aeefd8bf7" Oct 11 10:30:18.920535 master-1 kubenswrapper[4771]: I1011 10:30:18.920261 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-3-master-1" Oct 11 10:30:18.944092 master-1 kubenswrapper[4771]: I1011 10:30:18.944061 4771 scope.go:117] "RemoveContainer" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" Oct 11 10:30:18.945166 master-1 kubenswrapper[4771]: E1011 10:30:18.945062 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32\": container with ID starting with 5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32 not found: ID does not exist" containerID="5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32" Oct 11 10:30:18.945166 master-1 kubenswrapper[4771]: I1011 10:30:18.945150 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32"} err="failed to get container status \"5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32\": rpc error: code = NotFound desc = could not find container \"5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32\": container with ID starting with 5eb00d420609b3f327b546c9ccf510a7aca38ca978bdc995a30f1d5c6a5e3d32 not found: ID does not exist" Oct 11 10:30:18.948814 master-1 kubenswrapper[4771]: I1011 10:30:18.948758 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/806cd59c-056a-4fb4-a3b4-cb716c01cdea-tuning-conf-dir\") pod \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " Oct 11 10:30:18.949110 master-1 kubenswrapper[4771]: I1011 10:30:18.948820 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfgsk\" (UniqueName: \"kubernetes.io/projected/806cd59c-056a-4fb4-a3b4-cb716c01cdea-kube-api-access-tfgsk\") pod \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " Oct 11 10:30:18.949110 master-1 kubenswrapper[4771]: I1011 10:30:18.948879 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/806cd59c-056a-4fb4-a3b4-cb716c01cdea-cni-sysctl-allowlist\") pod \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " Oct 11 10:30:18.949110 master-1 kubenswrapper[4771]: I1011 10:30:18.948947 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/806cd59c-056a-4fb4-a3b4-cb716c01cdea-ready\") pod \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\" (UID: \"806cd59c-056a-4fb4-a3b4-cb716c01cdea\") " Oct 11 10:30:18.949930 master-1 kubenswrapper[4771]: I1011 10:30:18.949562 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/806cd59c-056a-4fb4-a3b4-cb716c01cdea-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "806cd59c-056a-4fb4-a3b4-cb716c01cdea" (UID: "806cd59c-056a-4fb4-a3b4-cb716c01cdea"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:30:18.950071 master-1 kubenswrapper[4771]: I1011 10:30:18.949834 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806cd59c-056a-4fb4-a3b4-cb716c01cdea-ready" (OuterVolumeSpecName: "ready") pod "806cd59c-056a-4fb4-a3b4-cb716c01cdea" (UID: "806cd59c-056a-4fb4-a3b4-cb716c01cdea"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:30:18.950591 master-1 kubenswrapper[4771]: I1011 10:30:18.950502 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806cd59c-056a-4fb4-a3b4-cb716c01cdea-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "806cd59c-056a-4fb4-a3b4-cb716c01cdea" (UID: "806cd59c-056a-4fb4-a3b4-cb716c01cdea"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:30:18.953283 master-1 kubenswrapper[4771]: I1011 10:30:18.953209 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806cd59c-056a-4fb4-a3b4-cb716c01cdea-kube-api-access-tfgsk" (OuterVolumeSpecName: "kube-api-access-tfgsk") pod "806cd59c-056a-4fb4-a3b4-cb716c01cdea" (UID: "806cd59c-056a-4fb4-a3b4-cb716c01cdea"). InnerVolumeSpecName "kube-api-access-tfgsk". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:30:19.050487 master-1 kubenswrapper[4771]: I1011 10:30:19.050403 4771 reconciler_common.go:293] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/806cd59c-056a-4fb4-a3b4-cb716c01cdea-tuning-conf-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:19.050487 master-1 kubenswrapper[4771]: I1011 10:30:19.050467 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-tfgsk\" (UniqueName: \"kubernetes.io/projected/806cd59c-056a-4fb4-a3b4-cb716c01cdea-kube-api-access-tfgsk\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:19.050487 master-1 kubenswrapper[4771]: I1011 10:30:19.050492 4771 reconciler_common.go:293] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/806cd59c-056a-4fb4-a3b4-cb716c01cdea-cni-sysctl-allowlist\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:19.050786 master-1 kubenswrapper[4771]: I1011 10:30:19.050516 4771 reconciler_common.go:293] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/806cd59c-056a-4fb4-a3b4-cb716c01cdea-ready\") on node \"master-1\" DevicePath \"\"" Oct 11 10:30:19.263553 master-1 kubenswrapper[4771]: I1011 10:30:19.263437 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-9d7j4"] Oct 11 10:30:19.266855 master-1 kubenswrapper[4771]: I1011 10:30:19.266795 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-9d7j4"] Oct 11 10:30:19.301741 master-1 kubenswrapper[4771]: I1011 10:30:19.301652 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5ddb89f76-z5t6x_04cd4a19-2532-43d1-9144-1f59d9e52d19/router/0.log" Oct 11 10:30:19.497788 master-1 kubenswrapper[4771]: I1011 10:30:19.497586 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:19.497788 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:19.497788 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:19.497788 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:19.497788 master-1 kubenswrapper[4771]: I1011 10:30:19.497730 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:19.902092 master-1 kubenswrapper[4771]: I1011 10:30:19.901956 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-3-master-1_776c3745-2f4c-4a78-b1cd-77a7a1532df3/installer/0.log" Oct 11 10:30:20.446056 master-1 kubenswrapper[4771]: I1011 10:30:20.445957 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="806cd59c-056a-4fb4-a3b4-cb716c01cdea" path="/var/lib/kubelet/pods/806cd59c-056a-4fb4-a3b4-cb716c01cdea/volumes" Oct 11 10:30:20.497304 master-1 kubenswrapper[4771]: I1011 10:30:20.497215 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:20.497304 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:20.497304 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:20.497304 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:20.497682 master-1 kubenswrapper[4771]: I1011 10:30:20.497305 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:20.503051 master-1 kubenswrapper[4771]: I1011 10:30:20.502903 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_installer-4-master-1_7662f87a-13ba-439c-b386-05e68284803c/installer/0.log" Oct 11 10:30:20.697107 master-1 kubenswrapper[4771]: I1011 10:30:20.696921 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-guard-master-1_bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe/guard/0.log" Oct 11 10:30:20.895564 master-1 kubenswrapper[4771]: I1011 10:30:20.895463 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_89fad8183e18ab3ad0c46d272335e5f8/wait-for-host-port/0.log" Oct 11 10:30:21.103136 master-1 kubenswrapper[4771]: I1011 10:30:21.103086 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_89fad8183e18ab3ad0c46d272335e5f8/kube-scheduler/0.log" Oct 11 10:30:21.298105 master-1 kubenswrapper[4771]: I1011 10:30:21.297948 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_89fad8183e18ab3ad0c46d272335e5f8/kube-scheduler-cert-syncer/0.log" Oct 11 10:30:21.497409 master-1 kubenswrapper[4771]: I1011 10:30:21.497141 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:21.497409 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:21.497409 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:21.497409 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:21.497409 master-1 kubenswrapper[4771]: I1011 10:30:21.497236 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:21.499790 master-1 kubenswrapper[4771]: I1011 10:30:21.499723 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_89fad8183e18ab3ad0c46d272335e5f8/kube-scheduler-recovery-controller/0.log" Oct 11 10:30:22.496680 master-1 kubenswrapper[4771]: I1011 10:30:22.496584 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:22.496680 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:22.496680 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:22.496680 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:22.496680 master-1 kubenswrapper[4771]: I1011 10:30:22.496673 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:23.500179 master-1 kubenswrapper[4771]: I1011 10:30:23.500071 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:23.500179 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:23.500179 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:23.500179 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:23.501800 master-1 kubenswrapper[4771]: I1011 10:30:23.500195 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:24.497165 master-1 kubenswrapper[4771]: I1011 10:30:24.497079 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-apiserver_apiserver-555f658fd6-n5n6g_027736d1-f3d3-490e-9ee1-d08bad7a25b7/fix-audit-permissions/0.log" Oct 11 10:30:24.498047 master-1 kubenswrapper[4771]: I1011 10:30:24.497961 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:24.498047 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:24.498047 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:24.498047 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:24.498405 master-1 kubenswrapper[4771]: I1011 10:30:24.498059 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:24.703529 master-1 kubenswrapper[4771]: I1011 10:30:24.703426 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-apiserver_apiserver-555f658fd6-n5n6g_027736d1-f3d3-490e-9ee1-d08bad7a25b7/openshift-apiserver/0.log" Oct 11 10:30:24.899618 master-1 kubenswrapper[4771]: I1011 10:30:24.899442 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-apiserver_apiserver-555f658fd6-n5n6g_027736d1-f3d3-490e-9ee1-d08bad7a25b7/openshift-apiserver-check-endpoints/0.log" Oct 11 10:30:25.497824 master-1 kubenswrapper[4771]: I1011 10:30:25.497733 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:25.497824 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:25.497824 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:25.497824 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:25.497824 master-1 kubenswrapper[4771]: I1011 10:30:25.497815 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:26.496987 master-1 kubenswrapper[4771]: I1011 10:30:26.496922 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:26.496987 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:26.496987 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:26.496987 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:26.498232 master-1 kubenswrapper[4771]: I1011 10:30:26.498175 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:27.496736 master-1 kubenswrapper[4771]: I1011 10:30:27.496397 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-operator-lifecycle-manager_collect-profiles-29336310-8nc4v_f4904d67-3c44-40d9-8ea8-026d727e9486/collect-profiles/0.log" Oct 11 10:30:27.497291 master-1 kubenswrapper[4771]: I1011 10:30:27.497234 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:27.497291 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:27.497291 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:27.497291 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:27.498195 master-1 kubenswrapper[4771]: I1011 10:30:27.497328 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:28.291598 master-1 kubenswrapper[4771]: I1011 10:30:28.291495 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-etcd/etcd-master-1" Oct 11 10:30:28.310102 master-1 kubenswrapper[4771]: I1011 10:30:28.310034 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-operator-lifecycle-manager_packageserver-77c85f5c6-6zxmm_68bdaf37-fa14-4c86-a697-881df7c9c7f1/packageserver/0.log" Oct 11 10:30:28.314519 master-1 kubenswrapper[4771]: I1011 10:30:28.314429 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-etcd/etcd-master-1" Oct 11 10:30:28.496970 master-1 kubenswrapper[4771]: I1011 10:30:28.496857 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:28.496970 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:28.496970 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:28.496970 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:28.497747 master-1 kubenswrapper[4771]: I1011 10:30:28.496998 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:29.497228 master-1 kubenswrapper[4771]: I1011 10:30:29.497123 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:29.497228 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:29.497228 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:29.497228 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:29.497228 master-1 kubenswrapper[4771]: I1011 10:30:29.497203 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:30.246144 master-1 kubenswrapper[4771]: I1011 10:30:30.246083 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:30:30.497027 master-1 kubenswrapper[4771]: I1011 10:30:30.496819 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:30.497027 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:30.497027 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:30.497027 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:30.497662 master-1 kubenswrapper[4771]: I1011 10:30:30.497588 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:31.497702 master-1 kubenswrapper[4771]: I1011 10:30:31.497606 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:31.497702 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:31.497702 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:31.497702 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:31.498427 master-1 kubenswrapper[4771]: I1011 10:30:31.497704 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:32.496551 master-1 kubenswrapper[4771]: I1011 10:30:32.496492 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:32.496551 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:32.496551 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:32.496551 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:32.496551 master-1 kubenswrapper[4771]: I1011 10:30:32.496552 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:33.027124 master-1 kubenswrapper[4771]: I1011 10:30:33.026967 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/installer-3-retry-1-master-1"] Oct 11 10:30:33.028271 master-1 kubenswrapper[4771]: E1011 10:30:33.027369 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="776c3745-2f4c-4a78-b1cd-77a7a1532df3" containerName="installer" Oct 11 10:30:33.028271 master-1 kubenswrapper[4771]: I1011 10:30:33.027432 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="776c3745-2f4c-4a78-b1cd-77a7a1532df3" containerName="installer" Oct 11 10:30:33.028271 master-1 kubenswrapper[4771]: E1011 10:30:33.027464 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="806cd59c-056a-4fb4-a3b4-cb716c01cdea" containerName="kube-multus-additional-cni-plugins" Oct 11 10:30:33.028271 master-1 kubenswrapper[4771]: I1011 10:30:33.027482 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="806cd59c-056a-4fb4-a3b4-cb716c01cdea" containerName="kube-multus-additional-cni-plugins" Oct 11 10:30:33.028271 master-1 kubenswrapper[4771]: E1011 10:30:33.027506 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="f4904d67-3c44-40d9-8ea8-026d727e9486" containerName="collect-profiles" Oct 11 10:30:33.028271 master-1 kubenswrapper[4771]: I1011 10:30:33.027523 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4904d67-3c44-40d9-8ea8-026d727e9486" containerName="collect-profiles" Oct 11 10:30:33.028271 master-1 kubenswrapper[4771]: I1011 10:30:33.027782 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="f4904d67-3c44-40d9-8ea8-026d727e9486" containerName="collect-profiles" Oct 11 10:30:33.028271 master-1 kubenswrapper[4771]: I1011 10:30:33.027808 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="806cd59c-056a-4fb4-a3b4-cb716c01cdea" containerName="kube-multus-additional-cni-plugins" Oct 11 10:30:33.028271 master-1 kubenswrapper[4771]: I1011 10:30:33.027832 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="776c3745-2f4c-4a78-b1cd-77a7a1532df3" containerName="installer" Oct 11 10:30:33.029028 master-1 kubenswrapper[4771]: I1011 10:30:33.028816 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.031856 master-1 kubenswrapper[4771]: I1011 10:30:33.031798 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-kubelet-dir\") pod \"installer-3-retry-1-master-1\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.032069 master-1 kubenswrapper[4771]: I1011 10:30:33.031884 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-var-lock\") pod \"installer-3-retry-1-master-1\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.032253 master-1 kubenswrapper[4771]: I1011 10:30:33.032028 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/041ccbf8-b64e-4909-b9ae-35b19705838a-kube-api-access\") pod \"installer-3-retry-1-master-1\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.033044 master-1 kubenswrapper[4771]: I1011 10:30:33.033006 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-controller-manager"/"kube-root-ca.crt" Oct 11 10:30:33.042011 master-1 kubenswrapper[4771]: I1011 10:30:33.041928 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-3-retry-1-master-1"] Oct 11 10:30:33.133116 master-1 kubenswrapper[4771]: I1011 10:30:33.133002 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-kubelet-dir\") pod \"installer-3-retry-1-master-1\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.133116 master-1 kubenswrapper[4771]: I1011 10:30:33.133107 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-var-lock\") pod \"installer-3-retry-1-master-1\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.133604 master-1 kubenswrapper[4771]: I1011 10:30:33.133137 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-kubelet-dir\") pod \"installer-3-retry-1-master-1\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.133604 master-1 kubenswrapper[4771]: I1011 10:30:33.133252 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/041ccbf8-b64e-4909-b9ae-35b19705838a-kube-api-access\") pod \"installer-3-retry-1-master-1\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.133604 master-1 kubenswrapper[4771]: I1011 10:30:33.133341 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-var-lock\") pod \"installer-3-retry-1-master-1\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.155987 master-1 kubenswrapper[4771]: I1011 10:30:33.155945 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/041ccbf8-b64e-4909-b9ae-35b19705838a-kube-api-access\") pod \"installer-3-retry-1-master-1\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.360561 master-1 kubenswrapper[4771]: I1011 10:30:33.360447 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:30:33.497980 master-1 kubenswrapper[4771]: I1011 10:30:33.497865 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:33.497980 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:33.497980 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:33.497980 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:33.497980 master-1 kubenswrapper[4771]: I1011 10:30:33.497950 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:33.652268 master-1 kubenswrapper[4771]: I1011 10:30:33.652046 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-3-retry-1-master-1"] Oct 11 10:30:33.661344 master-1 kubenswrapper[4771]: W1011 10:30:33.661257 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod041ccbf8_b64e_4909_b9ae_35b19705838a.slice/crio-50bab3edfacf7962918848de1e7a735ed5d3ec7f5b1d816706993472067c79b5 WatchSource:0}: Error finding container 50bab3edfacf7962918848de1e7a735ed5d3ec7f5b1d816706993472067c79b5: Status 404 returned error can't find the container with id 50bab3edfacf7962918848de1e7a735ed5d3ec7f5b1d816706993472067c79b5 Oct 11 10:30:34.010456 master-1 kubenswrapper[4771]: I1011 10:30:34.010326 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" event={"ID":"041ccbf8-b64e-4909-b9ae-35b19705838a","Type":"ContainerStarted","Data":"50bab3edfacf7962918848de1e7a735ed5d3ec7f5b1d816706993472067c79b5"} Oct 11 10:30:34.497742 master-1 kubenswrapper[4771]: I1011 10:30:34.497648 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:34.497742 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:34.497742 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:34.497742 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:34.498803 master-1 kubenswrapper[4771]: I1011 10:30:34.497761 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:34.876914 master-1 kubenswrapper[4771]: I1011 10:30:34.876834 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw"] Oct 11 10:30:34.877330 master-1 kubenswrapper[4771]: I1011 10:30:34.877274 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" containerID="cri-o://913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68" gracePeriod=120 Oct 11 10:30:35.018789 master-1 kubenswrapper[4771]: I1011 10:30:35.018723 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" event={"ID":"041ccbf8-b64e-4909-b9ae-35b19705838a","Type":"ContainerStarted","Data":"64183b8f0fe57cec48ea786bd6f2bde7521a6790010bcd3ba5698a2a91bb323f"} Oct 11 10:30:35.040460 master-1 kubenswrapper[4771]: I1011 10:30:35.040351 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" podStartSLOduration=2.040329303 podStartE2EDuration="2.040329303s" podCreationTimestamp="2025-10-11 10:30:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:30:35.036395356 +0000 UTC m=+267.010621817" watchObservedRunningTime="2025-10-11 10:30:35.040329303 +0000 UTC m=+267.014555764" Oct 11 10:30:35.497563 master-1 kubenswrapper[4771]: I1011 10:30:35.497475 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:35.497563 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:35.497563 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:35.497563 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:35.498266 master-1 kubenswrapper[4771]: I1011 10:30:35.497586 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:36.497003 master-1 kubenswrapper[4771]: I1011 10:30:36.496885 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:36.497003 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:36.497003 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:36.497003 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:36.497003 master-1 kubenswrapper[4771]: I1011 10:30:36.496988 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: I1011 10:30:37.348545 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:30:37.348665 master-1 kubenswrapper[4771]: I1011 10:30:37.348643 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:37.497240 master-1 kubenswrapper[4771]: I1011 10:30:37.497146 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:37.497240 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:37.497240 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:37.497240 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:37.497652 master-1 kubenswrapper[4771]: I1011 10:30:37.497260 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:38.497508 master-1 kubenswrapper[4771]: I1011 10:30:38.497428 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:38.497508 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:38.497508 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:38.497508 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:38.498591 master-1 kubenswrapper[4771]: I1011 10:30:38.497522 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:39.496792 master-1 kubenswrapper[4771]: I1011 10:30:39.496674 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:39.496792 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:39.496792 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:39.496792 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:39.496792 master-1 kubenswrapper[4771]: I1011 10:30:39.496764 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:40.497057 master-1 kubenswrapper[4771]: I1011 10:30:40.496951 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:40.497057 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:40.497057 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:40.497057 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:40.497057 master-1 kubenswrapper[4771]: I1011 10:30:40.497017 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:41.497636 master-1 kubenswrapper[4771]: I1011 10:30:41.497551 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:41.497636 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:41.497636 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:41.497636 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:41.498525 master-1 kubenswrapper[4771]: I1011 10:30:41.497637 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: I1011 10:30:42.349437 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:30:42.349561 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:30:42.350346 master-1 kubenswrapper[4771]: I1011 10:30:42.349566 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:42.497518 master-1 kubenswrapper[4771]: I1011 10:30:42.497427 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:42.497518 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:42.497518 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:42.497518 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:42.498392 master-1 kubenswrapper[4771]: I1011 10:30:42.497542 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:43.497582 master-1 kubenswrapper[4771]: I1011 10:30:43.497505 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:43.497582 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:43.497582 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:43.497582 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:43.497582 master-1 kubenswrapper[4771]: I1011 10:30:43.497584 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:44.497139 master-1 kubenswrapper[4771]: I1011 10:30:44.497016 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:44.497139 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:44.497139 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:44.497139 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:44.497139 master-1 kubenswrapper[4771]: I1011 10:30:44.497110 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:45.498531 master-1 kubenswrapper[4771]: I1011 10:30:45.498411 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:45.498531 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:45.498531 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:45.498531 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:45.498531 master-1 kubenswrapper[4771]: I1011 10:30:45.498511 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:46.498211 master-1 kubenswrapper[4771]: I1011 10:30:46.498097 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:46.498211 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:46.498211 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:46.498211 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:46.498211 master-1 kubenswrapper[4771]: I1011 10:30:46.498206 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:47.214481 master-1 kubenswrapper[4771]: I1011 10:30:47.214327 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: I1011 10:30:47.348164 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:30:47.348296 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:30:47.349005 master-1 kubenswrapper[4771]: I1011 10:30:47.348331 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:47.349005 master-1 kubenswrapper[4771]: I1011 10:30:47.348576 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:30:47.489119 master-1 kubenswrapper[4771]: I1011 10:30:47.488919 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ts25n"] Oct 11 10:30:47.489912 master-1 kubenswrapper[4771]: I1011 10:30:47.489854 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ts25n" Oct 11 10:30:47.493884 master-1 kubenswrapper[4771]: I1011 10:30:47.493805 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-ingress-canary"/"canary-serving-cert" Oct 11 10:30:47.493884 master-1 kubenswrapper[4771]: I1011 10:30:47.493854 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ingress-canary"/"kube-root-ca.crt" Oct 11 10:30:47.494270 master-1 kubenswrapper[4771]: I1011 10:30:47.494222 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-ingress-canary"/"openshift-service-ca.crt" Oct 11 10:30:47.496619 master-1 kubenswrapper[4771]: I1011 10:30:47.496543 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:47.496619 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:47.496619 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:47.496619 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:47.497034 master-1 kubenswrapper[4771]: I1011 10:30:47.496629 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:47.501435 master-1 kubenswrapper[4771]: I1011 10:30:47.501331 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ts25n"] Oct 11 10:30:47.522047 master-1 kubenswrapper[4771]: I1011 10:30:47.521026 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d1de2f-e159-4967-935f-e7227794e6b4-cert\") pod \"ingress-canary-ts25n\" (UID: \"11d1de2f-e159-4967-935f-e7227794e6b4\") " pod="openshift-ingress-canary/ingress-canary-ts25n" Oct 11 10:30:47.522047 master-1 kubenswrapper[4771]: I1011 10:30:47.521091 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsflr\" (UniqueName: \"kubernetes.io/projected/11d1de2f-e159-4967-935f-e7227794e6b4-kube-api-access-rsflr\") pod \"ingress-canary-ts25n\" (UID: \"11d1de2f-e159-4967-935f-e7227794e6b4\") " pod="openshift-ingress-canary/ingress-canary-ts25n" Oct 11 10:30:47.622799 master-1 kubenswrapper[4771]: I1011 10:30:47.622694 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d1de2f-e159-4967-935f-e7227794e6b4-cert\") pod \"ingress-canary-ts25n\" (UID: \"11d1de2f-e159-4967-935f-e7227794e6b4\") " pod="openshift-ingress-canary/ingress-canary-ts25n" Oct 11 10:30:47.622799 master-1 kubenswrapper[4771]: I1011 10:30:47.622778 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-rsflr\" (UniqueName: \"kubernetes.io/projected/11d1de2f-e159-4967-935f-e7227794e6b4-kube-api-access-rsflr\") pod \"ingress-canary-ts25n\" (UID: \"11d1de2f-e159-4967-935f-e7227794e6b4\") " pod="openshift-ingress-canary/ingress-canary-ts25n" Oct 11 10:30:47.623056 master-1 kubenswrapper[4771]: E1011 10:30:47.622991 4771 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Oct 11 10:30:47.623177 master-1 kubenswrapper[4771]: E1011 10:30:47.623128 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d1de2f-e159-4967-935f-e7227794e6b4-cert podName:11d1de2f-e159-4967-935f-e7227794e6b4 nodeName:}" failed. No retries permitted until 2025-10-11 10:30:48.123097497 +0000 UTC m=+280.097323968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/11d1de2f-e159-4967-935f-e7227794e6b4-cert") pod "ingress-canary-ts25n" (UID: "11d1de2f-e159-4967-935f-e7227794e6b4") : secret "canary-serving-cert" not found Oct 11 10:30:47.643710 master-1 kubenswrapper[4771]: I1011 10:30:47.643625 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsflr\" (UniqueName: \"kubernetes.io/projected/11d1de2f-e159-4967-935f-e7227794e6b4-kube-api-access-rsflr\") pod \"ingress-canary-ts25n\" (UID: \"11d1de2f-e159-4967-935f-e7227794e6b4\") " pod="openshift-ingress-canary/ingress-canary-ts25n" Oct 11 10:30:47.799497 master-1 kubenswrapper[4771]: I1011 10:30:47.799323 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:30:48.129722 master-1 kubenswrapper[4771]: I1011 10:30:48.129593 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d1de2f-e159-4967-935f-e7227794e6b4-cert\") pod \"ingress-canary-ts25n\" (UID: \"11d1de2f-e159-4967-935f-e7227794e6b4\") " pod="openshift-ingress-canary/ingress-canary-ts25n" Oct 11 10:30:48.135166 master-1 kubenswrapper[4771]: I1011 10:30:48.135053 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d1de2f-e159-4967-935f-e7227794e6b4-cert\") pod \"ingress-canary-ts25n\" (UID: \"11d1de2f-e159-4967-935f-e7227794e6b4\") " pod="openshift-ingress-canary/ingress-canary-ts25n" Oct 11 10:30:48.268611 master-1 kubenswrapper[4771]: E1011 10:30:48.268484 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" podUID="d7647696-42d9-4dd9-bc3b-a4d52a42cf9a" Oct 11 10:30:48.365042 master-1 kubenswrapper[4771]: E1011 10:30:48.364923 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" podUID="6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b" Oct 11 10:30:48.421640 master-1 kubenswrapper[4771]: I1011 10:30:48.421480 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ts25n" Oct 11 10:30:48.499750 master-1 kubenswrapper[4771]: I1011 10:30:48.499660 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:48.499750 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:48.499750 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:48.499750 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:48.500254 master-1 kubenswrapper[4771]: I1011 10:30:48.499757 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:48.878193 master-1 kubenswrapper[4771]: I1011 10:30:48.877663 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ts25n"] Oct 11 10:30:48.885123 master-1 kubenswrapper[4771]: W1011 10:30:48.884974 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d1de2f_e159_4967_935f_e7227794e6b4.slice/crio-04831bfcb220c1a21faa45b5bad9ab30e86939a4dfd17879df3441206774bf75 WatchSource:0}: Error finding container 04831bfcb220c1a21faa45b5bad9ab30e86939a4dfd17879df3441206774bf75: Status 404 returned error can't find the container with id 04831bfcb220c1a21faa45b5bad9ab30e86939a4dfd17879df3441206774bf75 Oct 11 10:30:49.096584 master-1 kubenswrapper[4771]: I1011 10:30:49.096505 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ts25n" event={"ID":"11d1de2f-e159-4967-935f-e7227794e6b4","Type":"ContainerStarted","Data":"04831bfcb220c1a21faa45b5bad9ab30e86939a4dfd17879df3441206774bf75"} Oct 11 10:30:49.096584 master-1 kubenswrapper[4771]: I1011 10:30:49.096547 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:30:49.096982 master-1 kubenswrapper[4771]: I1011 10:30:49.096632 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:30:49.497596 master-1 kubenswrapper[4771]: I1011 10:30:49.497493 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:49.497596 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:49.497596 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:49.497596 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:49.498042 master-1 kubenswrapper[4771]: I1011 10:30:49.497608 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:49.704000 master-1 kubenswrapper[4771]: I1011 10:30:49.703937 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/installer-5-master-1"] Oct 11 10:30:49.705104 master-1 kubenswrapper[4771]: I1011 10:30:49.705075 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:49.716121 master-1 kubenswrapper[4771]: I1011 10:30:49.716030 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-5-master-1"] Oct 11 10:30:49.750976 master-1 kubenswrapper[4771]: I1011 10:30:49.750763 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:49.750976 master-1 kubenswrapper[4771]: I1011 10:30:49.750932 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/fefe846e-30ef-4097-ac69-f771a74b2b98-kube-api-access\") pod \"installer-5-master-1\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:49.751324 master-1 kubenswrapper[4771]: I1011 10:30:49.751002 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-var-lock\") pod \"installer-5-master-1\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:49.853371 master-1 kubenswrapper[4771]: I1011 10:30:49.853283 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:49.853580 master-1 kubenswrapper[4771]: I1011 10:30:49.853408 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/fefe846e-30ef-4097-ac69-f771a74b2b98-kube-api-access\") pod \"installer-5-master-1\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:49.853580 master-1 kubenswrapper[4771]: I1011 10:30:49.853478 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-var-lock\") pod \"installer-5-master-1\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:49.853715 master-1 kubenswrapper[4771]: I1011 10:30:49.853615 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-var-lock\") pod \"installer-5-master-1\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:49.853715 master-1 kubenswrapper[4771]: I1011 10:30:49.853680 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:49.887903 master-1 kubenswrapper[4771]: I1011 10:30:49.887870 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/fefe846e-30ef-4097-ac69-f771a74b2b98-kube-api-access\") pod \"installer-5-master-1\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:50.031859 master-1 kubenswrapper[4771]: I1011 10:30:50.031716 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:30:50.484845 master-1 kubenswrapper[4771]: I1011 10:30:50.484761 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-5-master-1"] Oct 11 10:30:50.497140 master-1 kubenswrapper[4771]: I1011 10:30:50.497067 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:50.497140 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:50.497140 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:50.497140 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:50.497470 master-1 kubenswrapper[4771]: I1011 10:30:50.497160 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:51.026584 master-1 kubenswrapper[4771]: I1011 10:30:51.026491 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-apiserver/apiserver-555f658fd6-n5n6g"] Oct 11 10:30:51.027234 master-1 kubenswrapper[4771]: I1011 10:30:51.026890 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" containerID="cri-o://5ee744232b5a66fa90e18d0677b90fd7ff50cae1f9e1afc9158b036b712f32da" gracePeriod=120 Oct 11 10:30:51.027234 master-1 kubenswrapper[4771]: I1011 10:30:51.027075 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver-check-endpoints" containerID="cri-o://893d86a98f61447fa7f11deae879fe95aeccf34e5a1d5e59961a43c4a181ec43" gracePeriod=120 Oct 11 10:30:51.111044 master-1 kubenswrapper[4771]: I1011 10:30:51.110976 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-5-master-1" event={"ID":"fefe846e-30ef-4097-ac69-f771a74b2b98","Type":"ContainerStarted","Data":"c1459d1f4e5756af7a1eb4e6fee99340755fd65bcc4e328af030a32aa61bf860"} Oct 11 10:30:51.497880 master-1 kubenswrapper[4771]: I1011 10:30:51.497802 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:51.497880 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:51.497880 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:51.497880 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:51.497880 master-1 kubenswrapper[4771]: I1011 10:30:51.497875 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:52.119093 master-1 kubenswrapper[4771]: I1011 10:30:52.118975 4771 generic.go:334] "Generic (PLEG): container finished" podID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerID="893d86a98f61447fa7f11deae879fe95aeccf34e5a1d5e59961a43c4a181ec43" exitCode=0 Oct 11 10:30:52.119093 master-1 kubenswrapper[4771]: I1011 10:30:52.119056 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" event={"ID":"027736d1-f3d3-490e-9ee1-d08bad7a25b7","Type":"ContainerDied","Data":"893d86a98f61447fa7f11deae879fe95aeccf34e5a1d5e59961a43c4a181ec43"} Oct 11 10:30:52.120751 master-1 kubenswrapper[4771]: I1011 10:30:52.120684 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-5-master-1" event={"ID":"fefe846e-30ef-4097-ac69-f771a74b2b98","Type":"ContainerStarted","Data":"91d13e47f19b0473725a534d9929ad8d4221ea196c8d107ca009b7a28f766686"} Oct 11 10:30:52.123910 master-1 kubenswrapper[4771]: I1011 10:30:52.123827 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ts25n" event={"ID":"11d1de2f-e159-4967-935f-e7227794e6b4","Type":"ContainerStarted","Data":"103869ffe6a4e7707d5fbfc4f57248e30528d71a49060d3b2905fe7b63067371"} Oct 11 10:30:52.147926 master-1 kubenswrapper[4771]: I1011 10:30:52.147826 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/installer-5-master-1" podStartSLOduration=3.147807723 podStartE2EDuration="3.147807723s" podCreationTimestamp="2025-10-11 10:30:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:30:52.143155344 +0000 UTC m=+284.117381855" watchObservedRunningTime="2025-10-11 10:30:52.147807723 +0000 UTC m=+284.122034194" Oct 11 10:30:52.167865 master-1 kubenswrapper[4771]: I1011 10:30:52.167724 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ts25n" podStartSLOduration=2.924788827 podStartE2EDuration="5.167695663s" podCreationTimestamp="2025-10-11 10:30:47 +0000 UTC" firstStartedPulling="2025-10-11 10:30:48.887992205 +0000 UTC m=+280.862218686" lastFinishedPulling="2025-10-11 10:30:51.130899061 +0000 UTC m=+283.105125522" observedRunningTime="2025-10-11 10:30:52.165284572 +0000 UTC m=+284.139511053" watchObservedRunningTime="2025-10-11 10:30:52.167695663 +0000 UTC m=+284.141922104" Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: I1011 10:30:52.346931 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:30:52.347010 master-1 kubenswrapper[4771]: I1011 10:30:52.347006 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:52.496816 master-1 kubenswrapper[4771]: I1011 10:30:52.496659 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:52.496816 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:52.496816 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:52.496816 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:52.496816 master-1 kubenswrapper[4771]: I1011 10:30:52.496727 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:53.295657 master-1 kubenswrapper[4771]: I1011 10:30:53.295557 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:30:53.296545 master-1 kubenswrapper[4771]: E1011 10:30:53.295789 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:32:55.295755558 +0000 UTC m=+407.269982099 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:30:53.396961 master-1 kubenswrapper[4771]: I1011 10:30:53.396844 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:30:53.397615 master-1 kubenswrapper[4771]: E1011 10:30:53.397088 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:32:55.397050687 +0000 UTC m=+407.371277168 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:30:53.497547 master-1 kubenswrapper[4771]: I1011 10:30:53.497476 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:53.497547 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:53.497547 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:53.497547 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:53.498015 master-1 kubenswrapper[4771]: I1011 10:30:53.497572 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:54.497781 master-1 kubenswrapper[4771]: I1011 10:30:54.497690 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:54.497781 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:54.497781 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:54.497781 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:54.498867 master-1 kubenswrapper[4771]: I1011 10:30:54.497790 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: I1011 10:30:54.524440 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:30:54.524552 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:30:54.525630 master-1 kubenswrapper[4771]: I1011 10:30:54.524569 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:55.198724 master-1 kubenswrapper[4771]: E1011 10:30:55.198628 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" podUID="537a2b50-0394-47bd-941a-def350316943" Oct 11 10:30:55.498014 master-1 kubenswrapper[4771]: I1011 10:30:55.497772 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:55.498014 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:55.498014 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:55.498014 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:55.498014 master-1 kubenswrapper[4771]: I1011 10:30:55.497950 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:56.146040 master-1 kubenswrapper[4771]: I1011 10:30:56.145923 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:30:56.418458 master-1 kubenswrapper[4771]: I1011 10:30:56.418202 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-controller-manager/installer-3-retry-1-master-1"] Oct 11 10:30:56.418733 master-1 kubenswrapper[4771]: I1011 10:30:56.418597 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" podUID="041ccbf8-b64e-4909-b9ae-35b19705838a" containerName="installer" containerID="cri-o://64183b8f0fe57cec48ea786bd6f2bde7521a6790010bcd3ba5698a2a91bb323f" gracePeriod=30 Oct 11 10:30:56.497850 master-1 kubenswrapper[4771]: I1011 10:30:56.497771 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:56.497850 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:56.497850 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:56.497850 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:56.498793 master-1 kubenswrapper[4771]: I1011 10:30:56.497860 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: I1011 10:30:57.349321 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:30:57.349466 master-1 kubenswrapper[4771]: I1011 10:30:57.349425 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:57.497982 master-1 kubenswrapper[4771]: I1011 10:30:57.497865 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:57.497982 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:57.497982 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:57.497982 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:57.497982 master-1 kubenswrapper[4771]: I1011 10:30:57.497950 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:57.631251 master-1 kubenswrapper[4771]: E1011 10:30:57.631055 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" podUID="c9e9455e-0b47-4623-9b4c-ef79cf62a254" Oct 11 10:30:58.157914 master-1 kubenswrapper[4771]: I1011 10:30:58.157725 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:30:58.489637 master-1 kubenswrapper[4771]: I1011 10:30:58.489467 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/installer-1-master-1"] Oct 11 10:30:58.490450 master-1 kubenswrapper[4771]: I1011 10:30:58.490406 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.494499 master-1 kubenswrapper[4771]: I1011 10:30:58.493928 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-apiserver"/"kube-root-ca.crt" Oct 11 10:30:58.497418 master-1 kubenswrapper[4771]: I1011 10:30:58.497333 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:58.497418 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:58.497418 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:58.497418 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:58.497654 master-1 kubenswrapper[4771]: I1011 10:30:58.497466 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:58.502292 master-1 kubenswrapper[4771]: I1011 10:30:58.502185 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-1-master-1"] Oct 11 10:30:58.562845 master-1 kubenswrapper[4771]: I1011 10:30:58.562755 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-var-lock\") pod \"installer-1-master-1\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.563465 master-1 kubenswrapper[4771]: I1011 10:30:58.563432 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.563704 master-1 kubenswrapper[4771]: I1011 10:30:58.563675 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kube-api-access\") pod \"installer-1-master-1\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.665728 master-1 kubenswrapper[4771]: I1011 10:30:58.665590 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-var-lock\") pod \"installer-1-master-1\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.666431 master-1 kubenswrapper[4771]: I1011 10:30:58.665770 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.666431 master-1 kubenswrapper[4771]: I1011 10:30:58.665820 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-var-lock\") pod \"installer-1-master-1\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.666431 master-1 kubenswrapper[4771]: I1011 10:30:58.665867 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kube-api-access\") pod \"installer-1-master-1\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.666431 master-1 kubenswrapper[4771]: I1011 10:30:58.666007 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kubelet-dir\") pod \"installer-1-master-1\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.701763 master-1 kubenswrapper[4771]: I1011 10:30:58.701666 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kube-api-access\") pod \"installer-1-master-1\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.814810 master-1 kubenswrapper[4771]: I1011 10:30:58.814719 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:30:58.819498 master-1 kubenswrapper[4771]: I1011 10:30:58.819438 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/installer-4-master-1"] Oct 11 10:30:58.820450 master-1 kubenswrapper[4771]: I1011 10:30:58.820409 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:58.829664 master-1 kubenswrapper[4771]: I1011 10:30:58.829608 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-4-master-1"] Oct 11 10:30:58.868534 master-1 kubenswrapper[4771]: I1011 10:30:58.868451 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-kubelet-dir\") pod \"installer-4-master-1\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:58.868641 master-1 kubenswrapper[4771]: I1011 10:30:58.868582 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/e2d8f859-38d1-4916-8262-ff865eb9982c-kube-api-access\") pod \"installer-4-master-1\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:58.868792 master-1 kubenswrapper[4771]: I1011 10:30:58.868656 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-var-lock\") pod \"installer-4-master-1\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:58.970406 master-1 kubenswrapper[4771]: I1011 10:30:58.969986 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-kubelet-dir\") pod \"installer-4-master-1\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:58.970406 master-1 kubenswrapper[4771]: I1011 10:30:58.970111 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-kubelet-dir\") pod \"installer-4-master-1\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:58.970406 master-1 kubenswrapper[4771]: I1011 10:30:58.970119 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/e2d8f859-38d1-4916-8262-ff865eb9982c-kube-api-access\") pod \"installer-4-master-1\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:58.970406 master-1 kubenswrapper[4771]: I1011 10:30:58.970391 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-var-lock\") pod \"installer-4-master-1\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:58.970764 master-1 kubenswrapper[4771]: I1011 10:30:58.970492 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-var-lock\") pod \"installer-4-master-1\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:58.995239 master-1 kubenswrapper[4771]: I1011 10:30:58.995191 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/e2d8f859-38d1-4916-8262-ff865eb9982c-kube-api-access\") pod \"installer-4-master-1\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:59.183280 master-1 kubenswrapper[4771]: I1011 10:30:59.183070 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:30:59.300068 master-1 kubenswrapper[4771]: I1011 10:30:59.299596 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-1-master-1"] Oct 11 10:30:59.310187 master-1 kubenswrapper[4771]: W1011 10:30:59.310088 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf5e7e1ec_47a8_4283_9119_0d9d1343963e.slice/crio-59e5d21ded756cda1a7334faedc3327ece5481088be9442f57418e8a270fcabc WatchSource:0}: Error finding container 59e5d21ded756cda1a7334faedc3327ece5481088be9442f57418e8a270fcabc: Status 404 returned error can't find the container with id 59e5d21ded756cda1a7334faedc3327ece5481088be9442f57418e8a270fcabc Oct 11 10:30:59.497417 master-1 kubenswrapper[4771]: I1011 10:30:59.497287 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:30:59.497417 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:30:59.497417 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:30:59.497417 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:30:59.497417 master-1 kubenswrapper[4771]: I1011 10:30:59.497346 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: I1011 10:30:59.522470 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:30:59.522539 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:30:59.528446 master-1 kubenswrapper[4771]: I1011 10:30:59.522561 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:30:59.653093 master-1 kubenswrapper[4771]: I1011 10:30:59.653021 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-4-master-1"] Oct 11 10:30:59.719266 master-1 kubenswrapper[4771]: W1011 10:30:59.719211 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode2d8f859_38d1_4916_8262_ff865eb9982c.slice/crio-8a6f9bd6405227d24a21db358afae4da6b7dc87d9e61c80b59ee89918fb69c92 WatchSource:0}: Error finding container 8a6f9bd6405227d24a21db358afae4da6b7dc87d9e61c80b59ee89918fb69c92: Status 404 returned error can't find the container with id 8a6f9bd6405227d24a21db358afae4da6b7dc87d9e61c80b59ee89918fb69c92 Oct 11 10:31:00.169509 master-1 kubenswrapper[4771]: I1011 10:31:00.169440 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-1-master-1" event={"ID":"f5e7e1ec-47a8-4283-9119-0d9d1343963e","Type":"ContainerStarted","Data":"d38cc7e81ae0071969a185999498646cddc10ee8b65bed60da29b4c1f46a55dc"} Oct 11 10:31:00.169673 master-1 kubenswrapper[4771]: I1011 10:31:00.169516 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-1-master-1" event={"ID":"f5e7e1ec-47a8-4283-9119-0d9d1343963e","Type":"ContainerStarted","Data":"59e5d21ded756cda1a7334faedc3327ece5481088be9442f57418e8a270fcabc"} Oct 11 10:31:00.171755 master-1 kubenswrapper[4771]: I1011 10:31:00.171706 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-4-master-1" event={"ID":"e2d8f859-38d1-4916-8262-ff865eb9982c","Type":"ContainerStarted","Data":"0904aae89e47c25a2e93dd629d94914a7beb5e409d6b4e15ac6ddcfa1b57aa4d"} Oct 11 10:31:00.171860 master-1 kubenswrapper[4771]: I1011 10:31:00.171752 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-4-master-1" event={"ID":"e2d8f859-38d1-4916-8262-ff865eb9982c","Type":"ContainerStarted","Data":"8a6f9bd6405227d24a21db358afae4da6b7dc87d9e61c80b59ee89918fb69c92"} Oct 11 10:31:00.195283 master-1 kubenswrapper[4771]: I1011 10:31:00.195168 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/installer-1-master-1" podStartSLOduration=2.195143677 podStartE2EDuration="2.195143677s" podCreationTimestamp="2025-10-11 10:30:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:31:00.192977903 +0000 UTC m=+292.167204434" watchObservedRunningTime="2025-10-11 10:31:00.195143677 +0000 UTC m=+292.169370158" Oct 11 10:31:00.212689 master-1 kubenswrapper[4771]: I1011 10:31:00.212579 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/installer-4-master-1" podStartSLOduration=2.212553505 podStartE2EDuration="2.212553505s" podCreationTimestamp="2025-10-11 10:30:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:31:00.209479243 +0000 UTC m=+292.183705714" watchObservedRunningTime="2025-10-11 10:31:00.212553505 +0000 UTC m=+292.186779976" Oct 11 10:31:00.286634 master-1 kubenswrapper[4771]: I1011 10:31:00.286516 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:31:00.286909 master-1 kubenswrapper[4771]: E1011 10:31:00.286767 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:31:00.286989 master-1 kubenswrapper[4771]: E1011 10:31:00.286908 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:33:02.286873293 +0000 UTC m=+414.261099764 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:31:00.497330 master-1 kubenswrapper[4771]: I1011 10:31:00.497205 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:00.497330 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:00.497330 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:00.497330 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:00.497330 master-1 kubenswrapper[4771]: I1011 10:31:00.497307 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:01.498213 master-1 kubenswrapper[4771]: I1011 10:31:01.498105 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:01.498213 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:01.498213 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:01.498213 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:01.499051 master-1 kubenswrapper[4771]: I1011 10:31:01.498223 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: I1011 10:31:02.348748 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:02.348883 master-1 kubenswrapper[4771]: I1011 10:31:02.348854 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:02.496772 master-1 kubenswrapper[4771]: I1011 10:31:02.496651 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:02.496772 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:02.496772 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:02.496772 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:02.497288 master-1 kubenswrapper[4771]: I1011 10:31:02.496749 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:02.721909 master-1 kubenswrapper[4771]: I1011 10:31:02.721708 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:31:02.722718 master-1 kubenswrapper[4771]: E1011 10:31:02.721975 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:31:02.722718 master-1 kubenswrapper[4771]: E1011 10:31:02.722244 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:33:04.722213887 +0000 UTC m=+416.696440368 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:31:03.497027 master-1 kubenswrapper[4771]: I1011 10:31:03.496910 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:03.497027 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:03.497027 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:03.497027 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:03.497027 master-1 kubenswrapper[4771]: I1011 10:31:03.497003 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:04.496892 master-1 kubenswrapper[4771]: I1011 10:31:04.496785 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:04.496892 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:04.496892 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:04.496892 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:04.496892 master-1 kubenswrapper[4771]: I1011 10:31:04.496880 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: I1011 10:31:04.525061 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:04.525140 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:04.526437 master-1 kubenswrapper[4771]: I1011 10:31:04.525156 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:04.526437 master-1 kubenswrapper[4771]: I1011 10:31:04.525296 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:31:05.209230 master-1 kubenswrapper[4771]: I1011 10:31:05.209124 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-3-retry-1-master-1_041ccbf8-b64e-4909-b9ae-35b19705838a/installer/0.log" Oct 11 10:31:05.209476 master-1 kubenswrapper[4771]: I1011 10:31:05.209453 4771 generic.go:334] "Generic (PLEG): container finished" podID="041ccbf8-b64e-4909-b9ae-35b19705838a" containerID="64183b8f0fe57cec48ea786bd6f2bde7521a6790010bcd3ba5698a2a91bb323f" exitCode=1 Oct 11 10:31:05.209571 master-1 kubenswrapper[4771]: I1011 10:31:05.209554 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" event={"ID":"041ccbf8-b64e-4909-b9ae-35b19705838a","Type":"ContainerDied","Data":"64183b8f0fe57cec48ea786bd6f2bde7521a6790010bcd3ba5698a2a91bb323f"} Oct 11 10:31:05.443743 master-1 kubenswrapper[4771]: I1011 10:31:05.443668 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-3-retry-1-master-1_041ccbf8-b64e-4909-b9ae-35b19705838a/installer/0.log" Oct 11 10:31:05.444016 master-1 kubenswrapper[4771]: I1011 10:31:05.443788 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:31:05.456147 master-1 kubenswrapper[4771]: I1011 10:31:05.456081 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-kubelet-dir\") pod \"041ccbf8-b64e-4909-b9ae-35b19705838a\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " Oct 11 10:31:05.456461 master-1 kubenswrapper[4771]: I1011 10:31:05.456203 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "041ccbf8-b64e-4909-b9ae-35b19705838a" (UID: "041ccbf8-b64e-4909-b9ae-35b19705838a"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:05.456461 master-1 kubenswrapper[4771]: I1011 10:31:05.456247 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/041ccbf8-b64e-4909-b9ae-35b19705838a-kube-api-access\") pod \"041ccbf8-b64e-4909-b9ae-35b19705838a\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " Oct 11 10:31:05.456461 master-1 kubenswrapper[4771]: I1011 10:31:05.456302 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-var-lock\") pod \"041ccbf8-b64e-4909-b9ae-35b19705838a\" (UID: \"041ccbf8-b64e-4909-b9ae-35b19705838a\") " Oct 11 10:31:05.456461 master-1 kubenswrapper[4771]: I1011 10:31:05.456388 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-var-lock" (OuterVolumeSpecName: "var-lock") pod "041ccbf8-b64e-4909-b9ae-35b19705838a" (UID: "041ccbf8-b64e-4909-b9ae-35b19705838a"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:05.456757 master-1 kubenswrapper[4771]: I1011 10:31:05.456652 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:05.456757 master-1 kubenswrapper[4771]: I1011 10:31:05.456670 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/041ccbf8-b64e-4909-b9ae-35b19705838a-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:05.461530 master-1 kubenswrapper[4771]: I1011 10:31:05.461419 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041ccbf8-b64e-4909-b9ae-35b19705838a-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "041ccbf8-b64e-4909-b9ae-35b19705838a" (UID: "041ccbf8-b64e-4909-b9ae-35b19705838a"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:31:05.497414 master-1 kubenswrapper[4771]: I1011 10:31:05.497340 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:05.497414 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:05.497414 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:05.497414 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:05.498234 master-1 kubenswrapper[4771]: I1011 10:31:05.497441 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:05.557917 master-1 kubenswrapper[4771]: I1011 10:31:05.557879 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/041ccbf8-b64e-4909-b9ae-35b19705838a-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:06.220430 master-1 kubenswrapper[4771]: I1011 10:31:06.220334 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_installer-3-retry-1-master-1_041ccbf8-b64e-4909-b9ae-35b19705838a/installer/0.log" Oct 11 10:31:06.220879 master-1 kubenswrapper[4771]: I1011 10:31:06.220839 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" event={"ID":"041ccbf8-b64e-4909-b9ae-35b19705838a","Type":"ContainerDied","Data":"50bab3edfacf7962918848de1e7a735ed5d3ec7f5b1d816706993472067c79b5"} Oct 11 10:31:06.221070 master-1 kubenswrapper[4771]: I1011 10:31:06.220919 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-3-retry-1-master-1" Oct 11 10:31:06.221180 master-1 kubenswrapper[4771]: I1011 10:31:06.221035 4771 scope.go:117] "RemoveContainer" containerID="64183b8f0fe57cec48ea786bd6f2bde7521a6790010bcd3ba5698a2a91bb323f" Oct 11 10:31:06.268056 master-1 kubenswrapper[4771]: I1011 10:31:06.267941 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-controller-manager/installer-3-retry-1-master-1"] Oct 11 10:31:06.275160 master-1 kubenswrapper[4771]: I1011 10:31:06.275122 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-controller-manager/installer-3-retry-1-master-1"] Oct 11 10:31:06.447587 master-1 kubenswrapper[4771]: I1011 10:31:06.447477 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041ccbf8-b64e-4909-b9ae-35b19705838a" path="/var/lib/kubelet/pods/041ccbf8-b64e-4909-b9ae-35b19705838a/volumes" Oct 11 10:31:06.496969 master-1 kubenswrapper[4771]: I1011 10:31:06.496766 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:06.496969 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:06.496969 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:06.496969 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:06.496969 master-1 kubenswrapper[4771]: I1011 10:31:06.496866 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: I1011 10:31:07.345527 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:07.345613 master-1 kubenswrapper[4771]: I1011 10:31:07.345590 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:07.497342 master-1 kubenswrapper[4771]: I1011 10:31:07.497028 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:07.497342 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:07.497342 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:07.497342 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:07.497342 master-1 kubenswrapper[4771]: I1011 10:31:07.497136 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:08.497570 master-1 kubenswrapper[4771]: I1011 10:31:08.497433 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:08.497570 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:08.497570 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:08.497570 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:08.497570 master-1 kubenswrapper[4771]: I1011 10:31:08.497542 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:09.497868 master-1 kubenswrapper[4771]: I1011 10:31:09.497733 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:09.497868 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:09.497868 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:09.497868 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:09.499026 master-1 kubenswrapper[4771]: I1011 10:31:09.497862 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: I1011 10:31:09.522489 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:09.522571 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:09.523949 master-1 kubenswrapper[4771]: I1011 10:31:09.522581 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:10.497208 master-1 kubenswrapper[4771]: I1011 10:31:10.497070 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:10.497208 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:10.497208 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:10.497208 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:10.497208 master-1 kubenswrapper[4771]: I1011 10:31:10.497184 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:11.497407 master-1 kubenswrapper[4771]: I1011 10:31:11.497281 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:11.497407 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:11.497407 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:11.497407 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:11.497407 master-1 kubenswrapper[4771]: I1011 10:31:11.497403 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:12.282591 master-1 kubenswrapper[4771]: I1011 10:31:12.282517 4771 kubelet.go:1505] "Image garbage collection succeeded" Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: I1011 10:31:12.349059 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:12.349173 master-1 kubenswrapper[4771]: I1011 10:31:12.349152 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:12.497894 master-1 kubenswrapper[4771]: I1011 10:31:12.497538 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:12.497894 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:12.497894 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:12.497894 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:12.497894 master-1 kubenswrapper[4771]: I1011 10:31:12.497619 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:12.758056 master-1 kubenswrapper[4771]: I1011 10:31:12.757967 4771 patch_prober.go:28] interesting pod/machine-config-daemon-9nzpz container/machine-config-daemon namespace/openshift-machine-config-operator: Liveness probe status=failure output="Get \"http://127.0.0.1:8798/health\": dial tcp 127.0.0.1:8798: connect: connection refused" start-of-body= Oct 11 10:31:12.758465 master-1 kubenswrapper[4771]: I1011 10:31:12.758067 4771 prober.go:107] "Probe failed" probeType="Liveness" pod="openshift-machine-config-operator/machine-config-daemon-9nzpz" podUID="ebb73d72-cbb7-4736-870e-79e86c9fa7f5" containerName="machine-config-daemon" probeResult="failure" output="Get \"http://127.0.0.1:8798/health\": dial tcp 127.0.0.1:8798: connect: connection refused" Oct 11 10:31:13.496927 master-1 kubenswrapper[4771]: I1011 10:31:13.496835 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:13.496927 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:13.496927 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:13.496927 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:13.497412 master-1 kubenswrapper[4771]: I1011 10:31:13.496936 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:14.497679 master-1 kubenswrapper[4771]: I1011 10:31:14.497547 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:14.497679 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:14.497679 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:14.497679 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:14.497679 master-1 kubenswrapper[4771]: I1011 10:31:14.497660 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: I1011 10:31:14.524233 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:14.524308 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:14.525502 master-1 kubenswrapper[4771]: I1011 10:31:14.524326 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:15.497725 master-1 kubenswrapper[4771]: I1011 10:31:15.497607 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:15.497725 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:15.497725 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:15.497725 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:15.497725 master-1 kubenswrapper[4771]: I1011 10:31:15.497718 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:16.497384 master-1 kubenswrapper[4771]: I1011 10:31:16.497272 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:16.497384 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:16.497384 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:16.497384 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:16.498420 master-1 kubenswrapper[4771]: I1011 10:31:16.497416 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: I1011 10:31:17.349545 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:17.349679 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:17.350863 master-1 kubenswrapper[4771]: I1011 10:31:17.349685 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:17.498651 master-1 kubenswrapper[4771]: I1011 10:31:17.498554 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:17.498651 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:17.498651 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:17.498651 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:17.499767 master-1 kubenswrapper[4771]: I1011 10:31:17.498682 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:17.797585 master-1 kubenswrapper[4771]: I1011 10:31:17.797224 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:31:18.497847 master-1 kubenswrapper[4771]: I1011 10:31:18.497738 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:18.497847 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:18.497847 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:18.497847 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:18.498339 master-1 kubenswrapper[4771]: I1011 10:31:18.497840 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:19.496919 master-1 kubenswrapper[4771]: I1011 10:31:19.496846 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:19.496919 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:19.496919 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:19.496919 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:19.497679 master-1 kubenswrapper[4771]: I1011 10:31:19.497598 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: I1011 10:31:19.525176 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:19.525241 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:19.526085 master-1 kubenswrapper[4771]: I1011 10:31:19.525265 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:20.497669 master-1 kubenswrapper[4771]: I1011 10:31:20.497575 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:20.497669 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:20.497669 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:20.497669 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:20.498390 master-1 kubenswrapper[4771]: I1011 10:31:20.497671 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:21.497138 master-1 kubenswrapper[4771]: I1011 10:31:21.497012 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:21.497138 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:21.497138 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:21.497138 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:21.497138 master-1 kubenswrapper[4771]: I1011 10:31:21.497111 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: I1011 10:31:22.349261 4771 patch_prober.go:28] interesting pod/apiserver-65b6f4d4c9-skwvw container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:22.349351 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:22.350075 master-1 kubenswrapper[4771]: I1011 10:31:22.349386 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:22.497540 master-1 kubenswrapper[4771]: I1011 10:31:22.497450 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:22.497540 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:22.497540 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:22.497540 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:22.497540 master-1 kubenswrapper[4771]: I1011 10:31:22.497527 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:22.755419 master-1 kubenswrapper[4771]: I1011 10:31:22.755258 4771 kubelet.go:2431] "SyncLoop REMOVE" source="file" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:31:22.755839 master-1 kubenswrapper[4771]: I1011 10:31:22.755800 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler" containerID="cri-o://0400db595d18039edaf6ab7ccb3c1b1a3510ae9588fc33a6a91a15e993a6d1a4" gracePeriod=30 Oct 11 10:31:22.756049 master-1 kubenswrapper[4771]: I1011 10:31:22.755915 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler-recovery-controller" containerID="cri-o://4f12c3536caf37d890a386fecb2c94e5fc57775602e9a539771326b213c3ae7e" gracePeriod=30 Oct 11 10:31:22.756192 master-1 kubenswrapper[4771]: I1011 10:31:22.755989 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler-cert-syncer" containerID="cri-o://27a52449e5ec1bd52177b8ae4e5229c8bc4e5a7be149b07a0e7cb307be3932da" gracePeriod=30 Oct 11 10:31:22.756614 master-1 kubenswrapper[4771]: I1011 10:31:22.756558 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: E1011 10:31:22.756818 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="041ccbf8-b64e-4909-b9ae-35b19705838a" containerName="installer" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: I1011 10:31:22.756839 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="041ccbf8-b64e-4909-b9ae-35b19705838a" containerName="installer" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: E1011 10:31:22.756851 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: I1011 10:31:22.756860 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: E1011 10:31:22.756879 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="wait-for-host-port" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: I1011 10:31:22.756888 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="wait-for-host-port" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: E1011 10:31:22.756896 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler-recovery-controller" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: I1011 10:31:22.756903 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler-recovery-controller" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: E1011 10:31:22.756916 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler-cert-syncer" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: I1011 10:31:22.756924 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler-cert-syncer" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: I1011 10:31:22.757010 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="041ccbf8-b64e-4909-b9ae-35b19705838a" containerName="installer" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: I1011 10:31:22.757024 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler-cert-syncer" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: I1011 10:31:22.757036 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler" Oct 11 10:31:22.757314 master-1 kubenswrapper[4771]: I1011 10:31:22.757046 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="89fad8183e18ab3ad0c46d272335e5f8" containerName="kube-scheduler-recovery-controller" Oct 11 10:31:22.778012 master-1 kubenswrapper[4771]: I1011 10:31:22.777934 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-resource-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"a61df698d34d049669621b2249bfe758\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:22.778120 master-1 kubenswrapper[4771]: I1011 10:31:22.778046 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-cert-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"a61df698d34d049669621b2249bfe758\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:22.880433 master-1 kubenswrapper[4771]: I1011 10:31:22.880302 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-cert-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"a61df698d34d049669621b2249bfe758\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:22.880627 master-1 kubenswrapper[4771]: I1011 10:31:22.880562 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-resource-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"a61df698d34d049669621b2249bfe758\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:22.880742 master-1 kubenswrapper[4771]: I1011 10:31:22.880613 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-cert-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"a61df698d34d049669621b2249bfe758\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:22.880840 master-1 kubenswrapper[4771]: I1011 10:31:22.880672 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-resource-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"a61df698d34d049669621b2249bfe758\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:22.925861 master-1 kubenswrapper[4771]: I1011 10:31:22.925776 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_89fad8183e18ab3ad0c46d272335e5f8/kube-scheduler-cert-syncer/0.log" Oct 11 10:31:22.927221 master-1 kubenswrapper[4771]: I1011 10:31:22.927148 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:22.934320 master-1 kubenswrapper[4771]: I1011 10:31:22.934232 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="89fad8183e18ab3ad0c46d272335e5f8" podUID="a61df698d34d049669621b2249bfe758" Oct 11 10:31:22.981789 master-1 kubenswrapper[4771]: I1011 10:31:22.981686 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-cert-dir\") pod \"89fad8183e18ab3ad0c46d272335e5f8\" (UID: \"89fad8183e18ab3ad0c46d272335e5f8\") " Oct 11 10:31:22.982053 master-1 kubenswrapper[4771]: I1011 10:31:22.981880 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-resource-dir\") pod \"89fad8183e18ab3ad0c46d272335e5f8\" (UID: \"89fad8183e18ab3ad0c46d272335e5f8\") " Oct 11 10:31:22.982053 master-1 kubenswrapper[4771]: I1011 10:31:22.981914 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-cert-dir" (OuterVolumeSpecName: "cert-dir") pod "89fad8183e18ab3ad0c46d272335e5f8" (UID: "89fad8183e18ab3ad0c46d272335e5f8"). InnerVolumeSpecName "cert-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:22.982193 master-1 kubenswrapper[4771]: I1011 10:31:22.982087 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-resource-dir" (OuterVolumeSpecName: "resource-dir") pod "89fad8183e18ab3ad0c46d272335e5f8" (UID: "89fad8183e18ab3ad0c46d272335e5f8"). InnerVolumeSpecName "resource-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:22.982583 master-1 kubenswrapper[4771]: I1011 10:31:22.982541 4771 reconciler_common.go:293] "Volume detached for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-resource-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:22.982583 master-1 kubenswrapper[4771]: I1011 10:31:22.982578 4771 reconciler_common.go:293] "Volume detached for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/89fad8183e18ab3ad0c46d272335e5f8-cert-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:23.324692 master-1 kubenswrapper[4771]: I1011 10:31:23.324591 4771 generic.go:334] "Generic (PLEG): container finished" podID="fefe846e-30ef-4097-ac69-f771a74b2b98" containerID="91d13e47f19b0473725a534d9929ad8d4221ea196c8d107ca009b7a28f766686" exitCode=0 Oct 11 10:31:23.325004 master-1 kubenswrapper[4771]: I1011 10:31:23.324722 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-5-master-1" event={"ID":"fefe846e-30ef-4097-ac69-f771a74b2b98","Type":"ContainerDied","Data":"91d13e47f19b0473725a534d9929ad8d4221ea196c8d107ca009b7a28f766686"} Oct 11 10:31:23.327934 master-1 kubenswrapper[4771]: I1011 10:31:23.327861 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_89fad8183e18ab3ad0c46d272335e5f8/kube-scheduler-cert-syncer/0.log" Oct 11 10:31:23.329562 master-1 kubenswrapper[4771]: I1011 10:31:23.329511 4771 generic.go:334] "Generic (PLEG): container finished" podID="89fad8183e18ab3ad0c46d272335e5f8" containerID="4f12c3536caf37d890a386fecb2c94e5fc57775602e9a539771326b213c3ae7e" exitCode=0 Oct 11 10:31:23.329562 master-1 kubenswrapper[4771]: I1011 10:31:23.329549 4771 generic.go:334] "Generic (PLEG): container finished" podID="89fad8183e18ab3ad0c46d272335e5f8" containerID="27a52449e5ec1bd52177b8ae4e5229c8bc4e5a7be149b07a0e7cb307be3932da" exitCode=2 Oct 11 10:31:23.329562 master-1 kubenswrapper[4771]: I1011 10:31:23.329565 4771 generic.go:334] "Generic (PLEG): container finished" podID="89fad8183e18ab3ad0c46d272335e5f8" containerID="0400db595d18039edaf6ab7ccb3c1b1a3510ae9588fc33a6a91a15e993a6d1a4" exitCode=0 Oct 11 10:31:23.329813 master-1 kubenswrapper[4771]: I1011 10:31:23.329608 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bafca73396f947e9fa263ed96b26d1a45ed0144ffb97a2f796fec9628cf617b5" Oct 11 10:31:23.329813 master-1 kubenswrapper[4771]: I1011 10:31:23.329652 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:23.353458 master-1 kubenswrapper[4771]: I1011 10:31:23.353326 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="89fad8183e18ab3ad0c46d272335e5f8" podUID="a61df698d34d049669621b2249bfe758" Oct 11 10:31:23.361383 master-1 kubenswrapper[4771]: I1011 10:31:23.361296 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="89fad8183e18ab3ad0c46d272335e5f8" podUID="a61df698d34d049669621b2249bfe758" Oct 11 10:31:23.496315 master-1 kubenswrapper[4771]: I1011 10:31:23.496198 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:23.496315 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:23.496315 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:23.496315 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:23.496899 master-1 kubenswrapper[4771]: I1011 10:31:23.496325 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:24.450392 master-1 kubenswrapper[4771]: I1011 10:31:24.450295 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89fad8183e18ab3ad0c46d272335e5f8" path="/var/lib/kubelet/pods/89fad8183e18ab3ad0c46d272335e5f8/volumes" Oct 11 10:31:24.498052 master-1 kubenswrapper[4771]: I1011 10:31:24.497962 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:24.498052 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:24.498052 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:24.498052 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:24.498330 master-1 kubenswrapper[4771]: I1011 10:31:24.498050 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: I1011 10:31:24.524171 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:24.524285 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:24.525326 master-1 kubenswrapper[4771]: I1011 10:31:24.524330 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:24.760167 master-1 kubenswrapper[4771]: I1011 10:31:24.760090 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:31:24.806060 master-1 kubenswrapper[4771]: I1011 10:31:24.805991 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-kubelet-dir\") pod \"fefe846e-30ef-4097-ac69-f771a74b2b98\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " Oct 11 10:31:24.806310 master-1 kubenswrapper[4771]: I1011 10:31:24.806083 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/fefe846e-30ef-4097-ac69-f771a74b2b98-kube-api-access\") pod \"fefe846e-30ef-4097-ac69-f771a74b2b98\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " Oct 11 10:31:24.806310 master-1 kubenswrapper[4771]: I1011 10:31:24.806132 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-var-lock\") pod \"fefe846e-30ef-4097-ac69-f771a74b2b98\" (UID: \"fefe846e-30ef-4097-ac69-f771a74b2b98\") " Oct 11 10:31:24.806310 master-1 kubenswrapper[4771]: I1011 10:31:24.806189 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "fefe846e-30ef-4097-ac69-f771a74b2b98" (UID: "fefe846e-30ef-4097-ac69-f771a74b2b98"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:24.806643 master-1 kubenswrapper[4771]: I1011 10:31:24.806418 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-var-lock" (OuterVolumeSpecName: "var-lock") pod "fefe846e-30ef-4097-ac69-f771a74b2b98" (UID: "fefe846e-30ef-4097-ac69-f771a74b2b98"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:24.806830 master-1 kubenswrapper[4771]: I1011 10:31:24.806744 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:24.806930 master-1 kubenswrapper[4771]: I1011 10:31:24.806839 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/fefe846e-30ef-4097-ac69-f771a74b2b98-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:24.811151 master-1 kubenswrapper[4771]: I1011 10:31:24.811085 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefe846e-30ef-4097-ac69-f771a74b2b98-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "fefe846e-30ef-4097-ac69-f771a74b2b98" (UID: "fefe846e-30ef-4097-ac69-f771a74b2b98"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:31:24.908680 master-1 kubenswrapper[4771]: I1011 10:31:24.908566 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/fefe846e-30ef-4097-ac69-f771a74b2b98-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:25.341475 master-1 kubenswrapper[4771]: I1011 10:31:25.341386 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-5-master-1" event={"ID":"fefe846e-30ef-4097-ac69-f771a74b2b98","Type":"ContainerDied","Data":"c1459d1f4e5756af7a1eb4e6fee99340755fd65bcc4e328af030a32aa61bf860"} Oct 11 10:31:25.341475 master-1 kubenswrapper[4771]: I1011 10:31:25.341453 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-5-master-1" Oct 11 10:31:25.341475 master-1 kubenswrapper[4771]: I1011 10:31:25.341477 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1459d1f4e5756af7a1eb4e6fee99340755fd65bcc4e328af030a32aa61bf860" Oct 11 10:31:25.497947 master-1 kubenswrapper[4771]: I1011 10:31:25.497846 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:25.497947 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:25.497947 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:25.497947 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:25.497947 master-1 kubenswrapper[4771]: I1011 10:31:25.497936 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:26.336597 master-1 kubenswrapper[4771]: I1011 10:31:26.336523 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:31:26.347625 master-1 kubenswrapper[4771]: I1011 10:31:26.347543 4771 generic.go:334] "Generic (PLEG): container finished" podID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerID="913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68" exitCode=0 Oct 11 10:31:26.347625 master-1 kubenswrapper[4771]: I1011 10:31:26.347608 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" event={"ID":"004ee387-d0e9-4582-ad14-f571832ebd6e","Type":"ContainerDied","Data":"913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68"} Oct 11 10:31:26.347625 master-1 kubenswrapper[4771]: I1011 10:31:26.347618 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" Oct 11 10:31:26.347904 master-1 kubenswrapper[4771]: I1011 10:31:26.347648 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw" event={"ID":"004ee387-d0e9-4582-ad14-f571832ebd6e","Type":"ContainerDied","Data":"70ee09355a354a55a1e3cc86654a95e054448e4680cbf989813075d48bc93f03"} Oct 11 10:31:26.347904 master-1 kubenswrapper[4771]: I1011 10:31:26.347681 4771 scope.go:117] "RemoveContainer" containerID="913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68" Oct 11 10:31:26.365337 master-1 kubenswrapper[4771]: I1011 10:31:26.365296 4771 scope.go:117] "RemoveContainer" containerID="e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346" Oct 11 10:31:26.387020 master-1 kubenswrapper[4771]: I1011 10:31:26.386929 4771 scope.go:117] "RemoveContainer" containerID="913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68" Oct 11 10:31:26.387742 master-1 kubenswrapper[4771]: E1011 10:31:26.387687 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68\": container with ID starting with 913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68 not found: ID does not exist" containerID="913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68" Oct 11 10:31:26.387914 master-1 kubenswrapper[4771]: I1011 10:31:26.387753 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68"} err="failed to get container status \"913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68\": rpc error: code = NotFound desc = could not find container \"913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68\": container with ID starting with 913475a2d9db4e326668ab52c1e0002f7c3998815697a186b12ad2e219935d68 not found: ID does not exist" Oct 11 10:31:26.387914 master-1 kubenswrapper[4771]: I1011 10:31:26.387788 4771 scope.go:117] "RemoveContainer" containerID="e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346" Oct 11 10:31:26.388431 master-1 kubenswrapper[4771]: E1011 10:31:26.388319 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346\": container with ID starting with e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346 not found: ID does not exist" containerID="e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346" Oct 11 10:31:26.388431 master-1 kubenswrapper[4771]: I1011 10:31:26.388384 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346"} err="failed to get container status \"e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346\": rpc error: code = NotFound desc = could not find container \"e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346\": container with ID starting with e7797162c7d48146c8bfccf87f14747095a7d1d8794c4946a5c000f5385fa346 not found: ID does not exist" Oct 11 10:31:26.428163 master-1 kubenswrapper[4771]: I1011 10:31:26.428096 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-serving-ca\") pod \"004ee387-d0e9-4582-ad14-f571832ebd6e\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " Oct 11 10:31:26.428163 master-1 kubenswrapper[4771]: I1011 10:31:26.428196 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-client\") pod \"004ee387-d0e9-4582-ad14-f571832ebd6e\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " Oct 11 10:31:26.428601 master-1 kubenswrapper[4771]: I1011 10:31:26.428285 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-dir\") pod \"004ee387-d0e9-4582-ad14-f571832ebd6e\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " Oct 11 10:31:26.428601 master-1 kubenswrapper[4771]: I1011 10:31:26.428318 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-trusted-ca-bundle\") pod \"004ee387-d0e9-4582-ad14-f571832ebd6e\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " Oct 11 10:31:26.428601 master-1 kubenswrapper[4771]: I1011 10:31:26.428404 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-policies\") pod \"004ee387-d0e9-4582-ad14-f571832ebd6e\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " Oct 11 10:31:26.428601 master-1 kubenswrapper[4771]: I1011 10:31:26.428462 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrj8w\" (UniqueName: \"kubernetes.io/projected/004ee387-d0e9-4582-ad14-f571832ebd6e-kube-api-access-vrj8w\") pod \"004ee387-d0e9-4582-ad14-f571832ebd6e\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " Oct 11 10:31:26.428601 master-1 kubenswrapper[4771]: I1011 10:31:26.428477 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "004ee387-d0e9-4582-ad14-f571832ebd6e" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:26.428601 master-1 kubenswrapper[4771]: I1011 10:31:26.428526 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-encryption-config\") pod \"004ee387-d0e9-4582-ad14-f571832ebd6e\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " Oct 11 10:31:26.428601 master-1 kubenswrapper[4771]: I1011 10:31:26.428578 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert\") pod \"004ee387-d0e9-4582-ad14-f571832ebd6e\" (UID: \"004ee387-d0e9-4582-ad14-f571832ebd6e\") " Oct 11 10:31:26.429189 master-1 kubenswrapper[4771]: I1011 10:31:26.428894 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:26.429189 master-1 kubenswrapper[4771]: I1011 10:31:26.429149 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-serving-ca" (OuterVolumeSpecName: "etcd-serving-ca") pod "004ee387-d0e9-4582-ad14-f571832ebd6e" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e"). InnerVolumeSpecName "etcd-serving-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:31:26.429414 master-1 kubenswrapper[4771]: I1011 10:31:26.429312 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-policies" (OuterVolumeSpecName: "audit-policies") pod "004ee387-d0e9-4582-ad14-f571832ebd6e" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e"). InnerVolumeSpecName "audit-policies". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:31:26.429507 master-1 kubenswrapper[4771]: I1011 10:31:26.429470 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "004ee387-d0e9-4582-ad14-f571832ebd6e" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:31:26.432907 master-1 kubenswrapper[4771]: I1011 10:31:26.432840 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "004ee387-d0e9-4582-ad14-f571832ebd6e" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:31:26.433052 master-1 kubenswrapper[4771]: I1011 10:31:26.433018 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004ee387-d0e9-4582-ad14-f571832ebd6e-kube-api-access-vrj8w" (OuterVolumeSpecName: "kube-api-access-vrj8w") pod "004ee387-d0e9-4582-ad14-f571832ebd6e" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e"). InnerVolumeSpecName "kube-api-access-vrj8w". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:31:26.433804 master-1 kubenswrapper[4771]: I1011 10:31:26.433736 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-client" (OuterVolumeSpecName: "etcd-client") pod "004ee387-d0e9-4582-ad14-f571832ebd6e" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e"). InnerVolumeSpecName "etcd-client". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:31:26.434409 master-1 kubenswrapper[4771]: I1011 10:31:26.434279 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-encryption-config" (OuterVolumeSpecName: "encryption-config") pod "004ee387-d0e9-4582-ad14-f571832ebd6e" (UID: "004ee387-d0e9-4582-ad14-f571832ebd6e"). InnerVolumeSpecName "encryption-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:31:26.497672 master-1 kubenswrapper[4771]: I1011 10:31:26.497553 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:26.497672 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:26.497672 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:26.497672 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:26.498022 master-1 kubenswrapper[4771]: I1011 10:31:26.497676 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:26.530633 master-1 kubenswrapper[4771]: I1011 10:31:26.530482 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:26.530633 master-1 kubenswrapper[4771]: I1011 10:31:26.530540 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-serving-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:26.530633 master-1 kubenswrapper[4771]: I1011 10:31:26.530565 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-etcd-client\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:26.530633 master-1 kubenswrapper[4771]: I1011 10:31:26.530586 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:26.530633 master-1 kubenswrapper[4771]: I1011 10:31:26.530610 4771 reconciler_common.go:293] "Volume detached for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/004ee387-d0e9-4582-ad14-f571832ebd6e-audit-policies\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:26.530633 master-1 kubenswrapper[4771]: I1011 10:31:26.530629 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-vrj8w\" (UniqueName: \"kubernetes.io/projected/004ee387-d0e9-4582-ad14-f571832ebd6e-kube-api-access-vrj8w\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:26.531092 master-1 kubenswrapper[4771]: I1011 10:31:26.530649 4771 reconciler_common.go:293] "Volume detached for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/004ee387-d0e9-4582-ad14-f571832ebd6e-encryption-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:26.607157 master-1 kubenswrapper[4771]: I1011 10:31:26.607059 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:31:26.607491 master-1 kubenswrapper[4771]: I1011 10:31:26.607157 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:31:26.680640 master-1 kubenswrapper[4771]: I1011 10:31:26.680462 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw"] Oct 11 10:31:26.689523 master-1 kubenswrapper[4771]: I1011 10:31:26.689406 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-oauth-apiserver/apiserver-65b6f4d4c9-skwvw"] Oct 11 10:31:27.496831 master-1 kubenswrapper[4771]: I1011 10:31:27.496722 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:27.496831 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:27.496831 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:27.496831 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:27.497503 master-1 kubenswrapper[4771]: I1011 10:31:27.496837 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:28.446194 master-1 kubenswrapper[4771]: I1011 10:31:28.446114 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" path="/var/lib/kubelet/pods/004ee387-d0e9-4582-ad14-f571832ebd6e/volumes" Oct 11 10:31:28.498013 master-1 kubenswrapper[4771]: I1011 10:31:28.497935 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:28.498013 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:28.498013 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:28.498013 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:28.498424 master-1 kubenswrapper[4771]: I1011 10:31:28.498029 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:29.496982 master-1 kubenswrapper[4771]: I1011 10:31:29.496880 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:29.496982 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:29.496982 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:29.496982 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:29.496982 master-1 kubenswrapper[4771]: I1011 10:31:29.496959 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: I1011 10:31:29.524205 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:29.524332 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:29.525383 master-1 kubenswrapper[4771]: I1011 10:31:29.524410 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:30.497197 master-1 kubenswrapper[4771]: I1011 10:31:30.497097 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:31:30.497197 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:31:30.497197 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:31:30.497197 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:31:30.498498 master-1 kubenswrapper[4771]: I1011 10:31:30.497204 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:30.498498 master-1 kubenswrapper[4771]: I1011 10:31:30.497309 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:31:30.498498 master-1 kubenswrapper[4771]: I1011 10:31:30.498071 4771 kuberuntime_manager.go:1027] "Message for Container of pod" containerName="router" containerStatusID={"Type":"cri-o","ID":"d9d09acfb9b74efc71914e418c9f7ad84873a3a13515d6cfcddf159cfd555604"} pod="openshift-ingress/router-default-5ddb89f76-z5t6x" containerMessage="Container router failed startup probe, will be restarted" Oct 11 10:31:30.498498 master-1 kubenswrapper[4771]: I1011 10:31:30.498136 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" containerID="cri-o://d9d09acfb9b74efc71914e418c9f7ad84873a3a13515d6cfcddf159cfd555604" gracePeriod=3600 Oct 11 10:31:31.607092 master-1 kubenswrapper[4771]: I1011 10:31:31.606988 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:31:31.607092 master-1 kubenswrapper[4771]: I1011 10:31:31.607074 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: I1011 10:31:34.523308 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:34.523426 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:34.525541 master-1 kubenswrapper[4771]: I1011 10:31:34.523441 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:35.730911 master-1 kubenswrapper[4771]: I1011 10:31:35.730797 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk"] Oct 11 10:31:35.731738 master-1 kubenswrapper[4771]: E1011 10:31:35.731091 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="fefe846e-30ef-4097-ac69-f771a74b2b98" containerName="installer" Oct 11 10:31:35.731738 master-1 kubenswrapper[4771]: I1011 10:31:35.731114 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefe846e-30ef-4097-ac69-f771a74b2b98" containerName="installer" Oct 11 10:31:35.731738 master-1 kubenswrapper[4771]: E1011 10:31:35.731131 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" Oct 11 10:31:35.731738 master-1 kubenswrapper[4771]: I1011 10:31:35.731144 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" Oct 11 10:31:35.731738 master-1 kubenswrapper[4771]: E1011 10:31:35.731163 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="fix-audit-permissions" Oct 11 10:31:35.731738 master-1 kubenswrapper[4771]: I1011 10:31:35.731186 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="fix-audit-permissions" Oct 11 10:31:35.731738 master-1 kubenswrapper[4771]: I1011 10:31:35.731333 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="004ee387-d0e9-4582-ad14-f571832ebd6e" containerName="oauth-apiserver" Oct 11 10:31:35.731738 master-1 kubenswrapper[4771]: I1011 10:31:35.731386 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="fefe846e-30ef-4097-ac69-f771a74b2b98" containerName="installer" Oct 11 10:31:35.732284 master-1 kubenswrapper[4771]: I1011 10:31:35.732232 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.736195 master-1 kubenswrapper[4771]: I1011 10:31:35.736124 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"etcd-serving-ca" Oct 11 10:31:35.736912 master-1 kubenswrapper[4771]: I1011 10:31:35.736859 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"trusted-ca-bundle" Oct 11 10:31:35.737616 master-1 kubenswrapper[4771]: I1011 10:31:35.737491 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"openshift-service-ca.crt" Oct 11 10:31:35.737811 master-1 kubenswrapper[4771]: I1011 10:31:35.737678 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"etcd-client" Oct 11 10:31:35.737811 master-1 kubenswrapper[4771]: I1011 10:31:35.737784 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"audit-1" Oct 11 10:31:35.738172 master-1 kubenswrapper[4771]: I1011 10:31:35.737847 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"serving-cert" Oct 11 10:31:35.738172 master-1 kubenswrapper[4771]: I1011 10:31:35.737959 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"kube-root-ca.crt" Oct 11 10:31:35.738427 master-1 kubenswrapper[4771]: I1011 10:31:35.738235 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"encryption-config-1" Oct 11 10:31:35.751283 master-1 kubenswrapper[4771]: I1011 10:31:35.751216 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk"] Oct 11 10:31:35.852556 master-1 kubenswrapper[4771]: I1011 10:31:35.852437 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-serving-cert\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.852556 master-1 kubenswrapper[4771]: I1011 10:31:35.852524 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-serving-ca\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.852556 master-1 kubenswrapper[4771]: I1011 10:31:35.852569 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-trusted-ca-bundle\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.852957 master-1 kubenswrapper[4771]: I1011 10:31:35.852643 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-encryption-config\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.852957 master-1 kubenswrapper[4771]: I1011 10:31:35.852731 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kldk2\" (UniqueName: \"kubernetes.io/projected/d87cc032-b419-444c-8bf0-ef7405d7369d-kube-api-access-kldk2\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.852957 master-1 kubenswrapper[4771]: I1011 10:31:35.852799 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-dir\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.852957 master-1 kubenswrapper[4771]: I1011 10:31:35.852839 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-policies\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.853249 master-1 kubenswrapper[4771]: I1011 10:31:35.853012 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-client\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.954102 master-1 kubenswrapper[4771]: I1011 10:31:35.954013 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-client\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.954271 master-1 kubenswrapper[4771]: I1011 10:31:35.954104 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-serving-cert\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.954271 master-1 kubenswrapper[4771]: I1011 10:31:35.954142 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-serving-ca\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.954271 master-1 kubenswrapper[4771]: I1011 10:31:35.954166 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-trusted-ca-bundle\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.954271 master-1 kubenswrapper[4771]: I1011 10:31:35.954211 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-encryption-config\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.954271 master-1 kubenswrapper[4771]: I1011 10:31:35.954244 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-kldk2\" (UniqueName: \"kubernetes.io/projected/d87cc032-b419-444c-8bf0-ef7405d7369d-kube-api-access-kldk2\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.954613 master-1 kubenswrapper[4771]: I1011 10:31:35.954293 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-dir\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.954613 master-1 kubenswrapper[4771]: I1011 10:31:35.954321 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-policies\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.954987 master-1 kubenswrapper[4771]: I1011 10:31:35.954905 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-dir\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.955414 master-1 kubenswrapper[4771]: I1011 10:31:35.955332 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-policies\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.955639 master-1 kubenswrapper[4771]: I1011 10:31:35.955507 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-trusted-ca-bundle\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.956249 master-1 kubenswrapper[4771]: I1011 10:31:35.956159 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-serving-ca\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.959638 master-1 kubenswrapper[4771]: I1011 10:31:35.959566 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-encryption-config\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.959855 master-1 kubenswrapper[4771]: I1011 10:31:35.959702 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-client\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.960449 master-1 kubenswrapper[4771]: I1011 10:31:35.960344 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-serving-cert\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:35.988732 master-1 kubenswrapper[4771]: I1011 10:31:35.988588 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-kldk2\" (UniqueName: \"kubernetes.io/projected/d87cc032-b419-444c-8bf0-ef7405d7369d-kube-api-access-kldk2\") pod \"apiserver-6f855d6bcf-cwmmk\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:36.055825 master-1 kubenswrapper[4771]: I1011 10:31:36.055631 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:36.540436 master-1 kubenswrapper[4771]: I1011 10:31:36.540337 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk"] Oct 11 10:31:36.547590 master-1 kubenswrapper[4771]: W1011 10:31:36.547473 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87cc032_b419_444c_8bf0_ef7405d7369d.slice/crio-f8786873b90c54bfb0b515ad88ba2ef097b9f25b5ded48493272a640d89c1d55 WatchSource:0}: Error finding container f8786873b90c54bfb0b515ad88ba2ef097b9f25b5ded48493272a640d89c1d55: Status 404 returned error can't find the container with id f8786873b90c54bfb0b515ad88ba2ef097b9f25b5ded48493272a640d89c1d55 Oct 11 10:31:36.607736 master-1 kubenswrapper[4771]: I1011 10:31:36.607619 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:31:36.608059 master-1 kubenswrapper[4771]: I1011 10:31:36.607747 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:31:36.608059 master-1 kubenswrapper[4771]: I1011 10:31:36.607939 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:31:36.609568 master-1 kubenswrapper[4771]: I1011 10:31:36.609496 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:31:36.609728 master-1 kubenswrapper[4771]: I1011 10:31:36.609582 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:31:37.416393 master-1 kubenswrapper[4771]: I1011 10:31:37.416249 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" event={"ID":"d87cc032-b419-444c-8bf0-ef7405d7369d","Type":"ContainerDied","Data":"cc3604bd3c6d5088cac6e57645a1372932a2b915b7df557349ccea609bf9af52"} Oct 11 10:31:37.417531 master-1 kubenswrapper[4771]: I1011 10:31:37.417443 4771 generic.go:334] "Generic (PLEG): container finished" podID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerID="cc3604bd3c6d5088cac6e57645a1372932a2b915b7df557349ccea609bf9af52" exitCode=0 Oct 11 10:31:37.417655 master-1 kubenswrapper[4771]: I1011 10:31:37.417563 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" event={"ID":"d87cc032-b419-444c-8bf0-ef7405d7369d","Type":"ContainerStarted","Data":"f8786873b90c54bfb0b515ad88ba2ef097b9f25b5ded48493272a640d89c1d55"} Oct 11 10:31:38.049090 master-1 kubenswrapper[4771]: I1011 10:31:38.049007 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:31:38.051997 master-1 kubenswrapper[4771]: I1011 10:31:38.051551 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.082421 master-1 kubenswrapper[4771]: I1011 10:31:38.079082 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.082421 master-1 kubenswrapper[4771]: I1011 10:31:38.079535 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.082421 master-1 kubenswrapper[4771]: I1011 10:31:38.079609 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.099569 master-1 kubenswrapper[4771]: I1011 10:31:38.099514 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:31:38.181125 master-1 kubenswrapper[4771]: I1011 10:31:38.181042 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.181125 master-1 kubenswrapper[4771]: I1011 10:31:38.181108 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.181442 master-1 kubenswrapper[4771]: I1011 10:31:38.181171 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.181442 master-1 kubenswrapper[4771]: I1011 10:31:38.181242 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.181442 master-1 kubenswrapper[4771]: I1011 10:31:38.181239 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.181442 master-1 kubenswrapper[4771]: I1011 10:31:38.181339 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.395519 master-1 kubenswrapper[4771]: I1011 10:31:38.395288 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:38.421957 master-1 kubenswrapper[4771]: W1011 10:31:38.421912 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b1362996d1e0c2cea0bee73eb18468.slice/crio-8e5c43de2cc367ec1c4a7e349d6c7a564b4d6502ecb1a203dea90aba296e3027 WatchSource:0}: Error finding container 8e5c43de2cc367ec1c4a7e349d6c7a564b4d6502ecb1a203dea90aba296e3027: Status 404 returned error can't find the container with id 8e5c43de2cc367ec1c4a7e349d6c7a564b4d6502ecb1a203dea90aba296e3027 Oct 11 10:31:38.426228 master-1 kubenswrapper[4771]: I1011 10:31:38.426151 4771 generic.go:334] "Generic (PLEG): container finished" podID="f5e7e1ec-47a8-4283-9119-0d9d1343963e" containerID="d38cc7e81ae0071969a185999498646cddc10ee8b65bed60da29b4c1f46a55dc" exitCode=0 Oct 11 10:31:38.426416 master-1 kubenswrapper[4771]: I1011 10:31:38.426219 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-1-master-1" event={"ID":"f5e7e1ec-47a8-4283-9119-0d9d1343963e","Type":"ContainerDied","Data":"d38cc7e81ae0071969a185999498646cddc10ee8b65bed60da29b4c1f46a55dc"} Oct 11 10:31:38.429254 master-1 kubenswrapper[4771]: I1011 10:31:38.429189 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" event={"ID":"d87cc032-b419-444c-8bf0-ef7405d7369d","Type":"ContainerStarted","Data":"79f8e8a3af9681261cf6c96297e08774526c159a1df96245fda7d956c1a72204"} Oct 11 10:31:38.436125 master-1 kubenswrapper[4771]: I1011 10:31:38.436067 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:38.496925 master-1 kubenswrapper[4771]: I1011 10:31:38.496874 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="309f472e-b390-4e80-9837-cda7353ae2b9" Oct 11 10:31:38.497078 master-1 kubenswrapper[4771]: I1011 10:31:38.496931 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="309f472e-b390-4e80-9837-cda7353ae2b9" Oct 11 10:31:38.501667 master-1 kubenswrapper[4771]: I1011 10:31:38.501567 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podStartSLOduration=64.501539905 podStartE2EDuration="1m4.501539905s" podCreationTimestamp="2025-10-11 10:30:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:31:38.497178623 +0000 UTC m=+330.471405144" watchObservedRunningTime="2025-10-11 10:31:38.501539905 +0000 UTC m=+330.475766376" Oct 11 10:31:38.513426 master-1 kubenswrapper[4771]: I1011 10:31:38.513070 4771 kubelet.go:1914] "Deleted mirror pod because it is outdated" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:38.514782 master-1 kubenswrapper[4771]: I1011 10:31:38.514752 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:31:38.520219 master-1 kubenswrapper[4771]: I1011 10:31:38.520201 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:31:38.529447 master-1 kubenswrapper[4771]: I1011 10:31:38.529419 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:31:38.531789 master-1 kubenswrapper[4771]: I1011 10:31:38.531768 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:31:38.545083 master-1 kubenswrapper[4771]: W1011 10:31:38.545048 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61df698d34d049669621b2249bfe758.slice/crio-2d313c6886c969ebde302c0dbeabb72642e57f8da8af3701d11d8b20cfc8e2f6 WatchSource:0}: Error finding container 2d313c6886c969ebde302c0dbeabb72642e57f8da8af3701d11d8b20cfc8e2f6: Status 404 returned error can't find the container with id 2d313c6886c969ebde302c0dbeabb72642e57f8da8af3701d11d8b20cfc8e2f6 Oct 11 10:31:39.444841 master-1 kubenswrapper[4771]: I1011 10:31:39.444681 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"a61df698d34d049669621b2249bfe758","Type":"ContainerStarted","Data":"0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577"} Oct 11 10:31:39.444841 master-1 kubenswrapper[4771]: I1011 10:31:39.444745 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"a61df698d34d049669621b2249bfe758","Type":"ContainerStarted","Data":"2d313c6886c969ebde302c0dbeabb72642e57f8da8af3701d11d8b20cfc8e2f6"} Oct 11 10:31:39.447123 master-1 kubenswrapper[4771]: I1011 10:31:39.447060 4771 generic.go:334] "Generic (PLEG): container finished" podID="34b1362996d1e0c2cea0bee73eb18468" containerID="e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296" exitCode=0 Oct 11 10:31:39.447450 master-1 kubenswrapper[4771]: I1011 10:31:39.447400 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"34b1362996d1e0c2cea0bee73eb18468","Type":"ContainerDied","Data":"e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296"} Oct 11 10:31:39.447545 master-1 kubenswrapper[4771]: I1011 10:31:39.447454 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"34b1362996d1e0c2cea0bee73eb18468","Type":"ContainerStarted","Data":"8e5c43de2cc367ec1c4a7e349d6c7a564b4d6502ecb1a203dea90aba296e3027"} Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: I1011 10:31:39.521444 4771 patch_prober.go:28] interesting pod/apiserver-555f658fd6-n5n6g container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:31:39.522123 master-1 kubenswrapper[4771]: I1011 10:31:39.521524 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:31:39.781374 master-1 kubenswrapper[4771]: I1011 10:31:39.781302 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:31:39.802578 master-1 kubenswrapper[4771]: I1011 10:31:39.802494 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kubelet-dir\") pod \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " Oct 11 10:31:39.802772 master-1 kubenswrapper[4771]: I1011 10:31:39.802585 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-var-lock\") pod \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " Oct 11 10:31:39.802772 master-1 kubenswrapper[4771]: I1011 10:31:39.802657 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kube-api-access\") pod \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\" (UID: \"f5e7e1ec-47a8-4283-9119-0d9d1343963e\") " Oct 11 10:31:39.803513 master-1 kubenswrapper[4771]: I1011 10:31:39.803466 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "f5e7e1ec-47a8-4283-9119-0d9d1343963e" (UID: "f5e7e1ec-47a8-4283-9119-0d9d1343963e"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:39.803870 master-1 kubenswrapper[4771]: I1011 10:31:39.803823 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-var-lock" (OuterVolumeSpecName: "var-lock") pod "f5e7e1ec-47a8-4283-9119-0d9d1343963e" (UID: "f5e7e1ec-47a8-4283-9119-0d9d1343963e"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:39.806799 master-1 kubenswrapper[4771]: I1011 10:31:39.806718 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "f5e7e1ec-47a8-4283-9119-0d9d1343963e" (UID: "f5e7e1ec-47a8-4283-9119-0d9d1343963e"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:31:39.904611 master-1 kubenswrapper[4771]: I1011 10:31:39.904542 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:39.904785 master-1 kubenswrapper[4771]: I1011 10:31:39.904606 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f5e7e1ec-47a8-4283-9119-0d9d1343963e-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:39.904785 master-1 kubenswrapper[4771]: I1011 10:31:39.904692 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f5e7e1ec-47a8-4283-9119-0d9d1343963e-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:40.469957 master-1 kubenswrapper[4771]: I1011 10:31:40.468491 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-1-master-1" event={"ID":"f5e7e1ec-47a8-4283-9119-0d9d1343963e","Type":"ContainerDied","Data":"59e5d21ded756cda1a7334faedc3327ece5481088be9442f57418e8a270fcabc"} Oct 11 10:31:40.469957 master-1 kubenswrapper[4771]: I1011 10:31:40.468536 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e5d21ded756cda1a7334faedc3327ece5481088be9442f57418e8a270fcabc" Oct 11 10:31:40.469957 master-1 kubenswrapper[4771]: I1011 10:31:40.468602 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-1-master-1" Oct 11 10:31:40.477098 master-1 kubenswrapper[4771]: I1011 10:31:40.476969 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"34b1362996d1e0c2cea0bee73eb18468","Type":"ContainerStarted","Data":"d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10"} Oct 11 10:31:40.477098 master-1 kubenswrapper[4771]: I1011 10:31:40.477034 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"34b1362996d1e0c2cea0bee73eb18468","Type":"ContainerStarted","Data":"c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1"} Oct 11 10:31:40.477098 master-1 kubenswrapper[4771]: I1011 10:31:40.477048 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"34b1362996d1e0c2cea0bee73eb18468","Type":"ContainerStarted","Data":"7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea"} Oct 11 10:31:41.056024 master-1 kubenswrapper[4771]: I1011 10:31:41.055954 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:41.056024 master-1 kubenswrapper[4771]: I1011 10:31:41.056031 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:41.073229 master-1 kubenswrapper[4771]: I1011 10:31:41.073171 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:41.497211 master-1 kubenswrapper[4771]: I1011 10:31:41.497088 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"34b1362996d1e0c2cea0bee73eb18468","Type":"ContainerStarted","Data":"49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd"} Oct 11 10:31:41.497211 master-1 kubenswrapper[4771]: I1011 10:31:41.497144 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"34b1362996d1e0c2cea0bee73eb18468","Type":"ContainerStarted","Data":"3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7"} Oct 11 10:31:41.501922 master-1 kubenswrapper[4771]: I1011 10:31:41.501885 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:31:41.520227 master-1 kubenswrapper[4771]: I1011 10:31:41.520135 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/kube-apiserver-master-1" podStartSLOduration=3.520113094 podStartE2EDuration="3.520113094s" podCreationTimestamp="2025-10-11 10:31:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:31:41.518705972 +0000 UTC m=+333.492932433" watchObservedRunningTime="2025-10-11 10:31:41.520113094 +0000 UTC m=+333.494339545" Oct 11 10:31:41.607854 master-1 kubenswrapper[4771]: I1011 10:31:41.607785 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:31:41.608083 master-1 kubenswrapper[4771]: I1011 10:31:41.607879 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:31:42.507332 master-1 kubenswrapper[4771]: I1011 10:31:42.507100 4771 generic.go:334] "Generic (PLEG): container finished" podID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerID="5ee744232b5a66fa90e18d0677b90fd7ff50cae1f9e1afc9158b036b712f32da" exitCode=0 Oct 11 10:31:42.508879 master-1 kubenswrapper[4771]: I1011 10:31:42.508754 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" event={"ID":"027736d1-f3d3-490e-9ee1-d08bad7a25b7","Type":"ContainerDied","Data":"5ee744232b5a66fa90e18d0677b90fd7ff50cae1f9e1afc9158b036b712f32da"} Oct 11 10:31:42.518148 master-1 kubenswrapper[4771]: I1011 10:31:42.512281 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:42.630889 master-1 kubenswrapper[4771]: I1011 10:31:42.630788 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:31:42.740043 master-1 kubenswrapper[4771]: I1011 10:31:42.739969 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-serving-ca\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740043 master-1 kubenswrapper[4771]: I1011 10:31:42.740048 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-serving-cert\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740386 master-1 kubenswrapper[4771]: I1011 10:31:42.740087 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-config\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740386 master-1 kubenswrapper[4771]: I1011 10:31:42.740133 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-trusted-ca-bundle\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740386 master-1 kubenswrapper[4771]: I1011 10:31:42.740173 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740386 master-1 kubenswrapper[4771]: I1011 10:31:42.740206 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s269\" (UniqueName: \"kubernetes.io/projected/027736d1-f3d3-490e-9ee1-d08bad7a25b7-kube-api-access-4s269\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740386 master-1 kubenswrapper[4771]: I1011 10:31:42.740239 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit-dir\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740386 master-1 kubenswrapper[4771]: I1011 10:31:42.740289 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-encryption-config\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740386 master-1 kubenswrapper[4771]: I1011 10:31:42.740331 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-image-import-ca\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740834 master-1 kubenswrapper[4771]: I1011 10:31:42.740398 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-node-pullsecrets\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.740834 master-1 kubenswrapper[4771]: I1011 10:31:42.740435 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-client\") pod \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\" (UID: \"027736d1-f3d3-490e-9ee1-d08bad7a25b7\") " Oct 11 10:31:42.741233 master-1 kubenswrapper[4771]: I1011 10:31:42.741076 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-config" (OuterVolumeSpecName: "config") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:31:42.741233 master-1 kubenswrapper[4771]: I1011 10:31:42.741145 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-node-pullsecrets" (OuterVolumeSpecName: "node-pullsecrets") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "node-pullsecrets". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:42.741233 master-1 kubenswrapper[4771]: I1011 10:31:42.741127 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:42.741497 master-1 kubenswrapper[4771]: I1011 10:31:42.741233 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:31:42.741497 master-1 kubenswrapper[4771]: I1011 10:31:42.741473 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.741630 master-1 kubenswrapper[4771]: I1011 10:31:42.741503 4771 reconciler_common.go:293] "Volume detached for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/027736d1-f3d3-490e-9ee1-d08bad7a25b7-node-pullsecrets\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.741630 master-1 kubenswrapper[4771]: I1011 10:31:42.741578 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.741630 master-1 kubenswrapper[4771]: I1011 10:31:42.741598 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.741851 master-1 kubenswrapper[4771]: I1011 10:31:42.741635 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit" (OuterVolumeSpecName: "audit") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "audit". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:31:42.742595 master-1 kubenswrapper[4771]: I1011 10:31:42.742266 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-serving-ca" (OuterVolumeSpecName: "etcd-serving-ca") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "etcd-serving-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:31:42.742595 master-1 kubenswrapper[4771]: I1011 10:31:42.742510 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-image-import-ca" (OuterVolumeSpecName: "image-import-ca") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "image-import-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:31:42.744299 master-1 kubenswrapper[4771]: I1011 10:31:42.744237 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:31:42.744492 master-1 kubenswrapper[4771]: I1011 10:31:42.744432 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-encryption-config" (OuterVolumeSpecName: "encryption-config") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "encryption-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:31:42.744745 master-1 kubenswrapper[4771]: I1011 10:31:42.744697 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-client" (OuterVolumeSpecName: "etcd-client") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "etcd-client". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:31:42.745605 master-1 kubenswrapper[4771]: I1011 10:31:42.745502 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027736d1-f3d3-490e-9ee1-d08bad7a25b7-kube-api-access-4s269" (OuterVolumeSpecName: "kube-api-access-4s269") pod "027736d1-f3d3-490e-9ee1-d08bad7a25b7" (UID: "027736d1-f3d3-490e-9ee1-d08bad7a25b7"). InnerVolumeSpecName "kube-api-access-4s269". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:31:42.842469 master-1 kubenswrapper[4771]: I1011 10:31:42.842339 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-client\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.842469 master-1 kubenswrapper[4771]: I1011 10:31:42.842414 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-etcd-serving-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.842469 master-1 kubenswrapper[4771]: I1011 10:31:42.842423 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.842469 master-1 kubenswrapper[4771]: I1011 10:31:42.842432 4771 reconciler_common.go:293] "Volume detached for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-audit\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.842469 master-1 kubenswrapper[4771]: I1011 10:31:42.842444 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-4s269\" (UniqueName: \"kubernetes.io/projected/027736d1-f3d3-490e-9ee1-d08bad7a25b7-kube-api-access-4s269\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.842469 master-1 kubenswrapper[4771]: I1011 10:31:42.842456 4771 reconciler_common.go:293] "Volume detached for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/027736d1-f3d3-490e-9ee1-d08bad7a25b7-encryption-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.842469 master-1 kubenswrapper[4771]: I1011 10:31:42.842469 4771 reconciler_common.go:293] "Volume detached for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/027736d1-f3d3-490e-9ee1-d08bad7a25b7-image-import-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:42.911296 master-1 kubenswrapper[4771]: I1011 10:31:42.911222 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/kube-apiserver-guard-master-1"] Oct 11 10:31:42.911590 master-1 kubenswrapper[4771]: E1011 10:31:42.911481 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="f5e7e1ec-47a8-4283-9119-0d9d1343963e" containerName="installer" Oct 11 10:31:42.911590 master-1 kubenswrapper[4771]: I1011 10:31:42.911501 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e7e1ec-47a8-4283-9119-0d9d1343963e" containerName="installer" Oct 11 10:31:42.911590 master-1 kubenswrapper[4771]: E1011 10:31:42.911519 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" Oct 11 10:31:42.911590 master-1 kubenswrapper[4771]: I1011 10:31:42.911530 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" Oct 11 10:31:42.911590 master-1 kubenswrapper[4771]: E1011 10:31:42.911553 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="fix-audit-permissions" Oct 11 10:31:42.911590 master-1 kubenswrapper[4771]: I1011 10:31:42.911561 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="fix-audit-permissions" Oct 11 10:31:42.911590 master-1 kubenswrapper[4771]: E1011 10:31:42.911572 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver-check-endpoints" Oct 11 10:31:42.911590 master-1 kubenswrapper[4771]: I1011 10:31:42.911580 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver-check-endpoints" Oct 11 10:31:42.912025 master-1 kubenswrapper[4771]: I1011 10:31:42.911684 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver" Oct 11 10:31:42.912025 master-1 kubenswrapper[4771]: I1011 10:31:42.911700 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" containerName="openshift-apiserver-check-endpoints" Oct 11 10:31:42.912025 master-1 kubenswrapper[4771]: I1011 10:31:42.911720 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="f5e7e1ec-47a8-4283-9119-0d9d1343963e" containerName="installer" Oct 11 10:31:42.912306 master-1 kubenswrapper[4771]: I1011 10:31:42.912253 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:31:42.915551 master-1 kubenswrapper[4771]: I1011 10:31:42.915506 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-apiserver"/"openshift-service-ca.crt" Oct 11 10:31:42.915680 master-1 kubenswrapper[4771]: I1011 10:31:42.915550 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-apiserver"/"kube-root-ca.crt" Oct 11 10:31:42.919525 master-1 kubenswrapper[4771]: I1011 10:31:42.919470 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/kube-apiserver-guard-master-1"] Oct 11 10:31:42.944098 master-1 kubenswrapper[4771]: I1011 10:31:42.943636 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxxx\" (UniqueName: \"kubernetes.io/projected/86b914fa-4ccd-42fb-965a-a1bc19442489-kube-api-access-pkxxx\") pod \"kube-apiserver-guard-master-1\" (UID: \"86b914fa-4ccd-42fb-965a-a1bc19442489\") " pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:31:43.044809 master-1 kubenswrapper[4771]: I1011 10:31:43.044699 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxxx\" (UniqueName: \"kubernetes.io/projected/86b914fa-4ccd-42fb-965a-a1bc19442489-kube-api-access-pkxxx\") pod \"kube-apiserver-guard-master-1\" (UID: \"86b914fa-4ccd-42fb-965a-a1bc19442489\") " pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:31:43.065094 master-1 kubenswrapper[4771]: I1011 10:31:43.065011 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxxx\" (UniqueName: \"kubernetes.io/projected/86b914fa-4ccd-42fb-965a-a1bc19442489-kube-api-access-pkxxx\") pod \"kube-apiserver-guard-master-1\" (UID: \"86b914fa-4ccd-42fb-965a-a1bc19442489\") " pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:31:43.237185 master-1 kubenswrapper[4771]: I1011 10:31:43.236936 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:31:43.262882 master-1 kubenswrapper[4771]: I1011 10:31:43.262805 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:31:43.264681 master-1 kubenswrapper[4771]: I1011 10:31:43.264628 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:43.347748 master-1 kubenswrapper[4771]: I1011 10:31:43.347018 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:31:43.348905 master-1 kubenswrapper[4771]: I1011 10:31:43.348867 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-cert-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0c6dd9eb5bc384e5fbc388e7a2f95c28\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:43.349138 master-1 kubenswrapper[4771]: I1011 10:31:43.349111 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-resource-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0c6dd9eb5bc384e5fbc388e7a2f95c28\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:43.395599 master-1 kubenswrapper[4771]: I1011 10:31:43.395533 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:43.395801 master-1 kubenswrapper[4771]: I1011 10:31:43.395627 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:43.405060 master-1 kubenswrapper[4771]: I1011 10:31:43.404997 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:43.451156 master-1 kubenswrapper[4771]: I1011 10:31:43.451068 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-resource-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0c6dd9eb5bc384e5fbc388e7a2f95c28\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:43.451388 master-1 kubenswrapper[4771]: I1011 10:31:43.451226 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-cert-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0c6dd9eb5bc384e5fbc388e7a2f95c28\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:43.451388 master-1 kubenswrapper[4771]: I1011 10:31:43.451238 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-resource-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0c6dd9eb5bc384e5fbc388e7a2f95c28\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:43.452028 master-1 kubenswrapper[4771]: I1011 10:31:43.451962 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-cert-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0c6dd9eb5bc384e5fbc388e7a2f95c28\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:43.515505 master-1 kubenswrapper[4771]: I1011 10:31:43.515286 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" event={"ID":"027736d1-f3d3-490e-9ee1-d08bad7a25b7","Type":"ContainerDied","Data":"0245a7fd6940eab125c14495c22d9aa4a273c8034b951fafcde945d3497b7a29"} Oct 11 10:31:43.515505 master-1 kubenswrapper[4771]: I1011 10:31:43.515428 4771 scope.go:117] "RemoveContainer" containerID="893d86a98f61447fa7f11deae879fe95aeccf34e5a1d5e59961a43c4a181ec43" Oct 11 10:31:43.515505 master-1 kubenswrapper[4771]: I1011 10:31:43.515310 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-555f658fd6-n5n6g" Oct 11 10:31:43.517843 master-1 kubenswrapper[4771]: I1011 10:31:43.517787 4771 generic.go:334] "Generic (PLEG): container finished" podID="e2d8f859-38d1-4916-8262-ff865eb9982c" containerID="0904aae89e47c25a2e93dd629d94914a7beb5e409d6b4e15ac6ddcfa1b57aa4d" exitCode=0 Oct 11 10:31:43.517936 master-1 kubenswrapper[4771]: I1011 10:31:43.517869 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-4-master-1" event={"ID":"e2d8f859-38d1-4916-8262-ff865eb9982c","Type":"ContainerDied","Data":"0904aae89e47c25a2e93dd629d94914a7beb5e409d6b4e15ac6ddcfa1b57aa4d"} Oct 11 10:31:43.523266 master-1 kubenswrapper[4771]: I1011 10:31:43.523207 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:31:43.550058 master-1 kubenswrapper[4771]: I1011 10:31:43.549999 4771 scope.go:117] "RemoveContainer" containerID="5ee744232b5a66fa90e18d0677b90fd7ff50cae1f9e1afc9158b036b712f32da" Oct 11 10:31:43.570344 master-1 kubenswrapper[4771]: I1011 10:31:43.570279 4771 scope.go:117] "RemoveContainer" containerID="41af63c058a1e7b90357082e0adac794e0e1b2996f71cfa6b9c3a91b7079c8d7" Oct 11 10:31:43.596646 master-1 kubenswrapper[4771]: I1011 10:31:43.596581 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-apiserver/apiserver-555f658fd6-n5n6g"] Oct 11 10:31:43.602821 master-1 kubenswrapper[4771]: I1011 10:31:43.602772 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-apiserver/apiserver-555f658fd6-n5n6g"] Oct 11 10:31:43.645212 master-1 kubenswrapper[4771]: I1011 10:31:43.645130 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:43.662133 master-1 kubenswrapper[4771]: W1011 10:31:43.662069 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6dd9eb5bc384e5fbc388e7a2f95c28.slice/crio-c4fbba2242ad7533417c9230aa1cf12834880759f22bbf109485cc2aad57f7e7 WatchSource:0}: Error finding container c4fbba2242ad7533417c9230aa1cf12834880759f22bbf109485cc2aad57f7e7: Status 404 returned error can't find the container with id c4fbba2242ad7533417c9230aa1cf12834880759f22bbf109485cc2aad57f7e7 Oct 11 10:31:43.703409 master-1 kubenswrapper[4771]: I1011 10:31:43.703345 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/kube-apiserver-guard-master-1"] Oct 11 10:31:43.713411 master-1 kubenswrapper[4771]: W1011 10:31:43.713371 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b914fa_4ccd_42fb_965a_a1bc19442489.slice/crio-8732f5bab9a381941e5c31ffdd4f43b464797523fb9db751b3a1d2f56fdfdd7f WatchSource:0}: Error finding container 8732f5bab9a381941e5c31ffdd4f43b464797523fb9db751b3a1d2f56fdfdd7f: Status 404 returned error can't find the container with id 8732f5bab9a381941e5c31ffdd4f43b464797523fb9db751b3a1d2f56fdfdd7f Oct 11 10:31:43.940266 master-1 kubenswrapper[4771]: I1011 10:31:43.940175 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 10:31:44.446270 master-1 kubenswrapper[4771]: I1011 10:31:44.446200 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="027736d1-f3d3-490e-9ee1-d08bad7a25b7" path="/var/lib/kubelet/pods/027736d1-f3d3-490e-9ee1-d08bad7a25b7/volumes" Oct 11 10:31:44.523660 master-1 kubenswrapper[4771]: I1011 10:31:44.523614 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" event={"ID":"86b914fa-4ccd-42fb-965a-a1bc19442489","Type":"ContainerStarted","Data":"16917522dd8d6c564900b32768804f2aa15698ba6c7c0d38122dff408276f3fa"} Oct 11 10:31:44.524282 master-1 kubenswrapper[4771]: I1011 10:31:44.524257 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:31:44.524421 master-1 kubenswrapper[4771]: I1011 10:31:44.524402 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" event={"ID":"86b914fa-4ccd-42fb-965a-a1bc19442489","Type":"ContainerStarted","Data":"8732f5bab9a381941e5c31ffdd4f43b464797523fb9db751b3a1d2f56fdfdd7f"} Oct 11 10:31:44.525758 master-1 kubenswrapper[4771]: I1011 10:31:44.525703 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0c6dd9eb5bc384e5fbc388e7a2f95c28","Type":"ContainerStarted","Data":"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db"} Oct 11 10:31:44.525853 master-1 kubenswrapper[4771]: I1011 10:31:44.525782 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0c6dd9eb5bc384e5fbc388e7a2f95c28","Type":"ContainerStarted","Data":"c4fbba2242ad7533417c9230aa1cf12834880759f22bbf109485cc2aad57f7e7"} Oct 11 10:31:44.530382 master-1 kubenswrapper[4771]: I1011 10:31:44.530343 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:31:44.542916 master-1 kubenswrapper[4771]: I1011 10:31:44.542871 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podStartSLOduration=2.542857399 podStartE2EDuration="2.542857399s" podCreationTimestamp="2025-10-11 10:31:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:31:44.541142447 +0000 UTC m=+336.515368898" watchObservedRunningTime="2025-10-11 10:31:44.542857399 +0000 UTC m=+336.517083850" Oct 11 10:31:44.878338 master-1 kubenswrapper[4771]: I1011 10:31:44.878285 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:31:44.971438 master-1 kubenswrapper[4771]: I1011 10:31:44.971335 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-kubelet-dir\") pod \"e2d8f859-38d1-4916-8262-ff865eb9982c\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " Oct 11 10:31:44.971637 master-1 kubenswrapper[4771]: I1011 10:31:44.971443 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/e2d8f859-38d1-4916-8262-ff865eb9982c-kube-api-access\") pod \"e2d8f859-38d1-4916-8262-ff865eb9982c\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " Oct 11 10:31:44.971637 master-1 kubenswrapper[4771]: I1011 10:31:44.971477 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-var-lock\") pod \"e2d8f859-38d1-4916-8262-ff865eb9982c\" (UID: \"e2d8f859-38d1-4916-8262-ff865eb9982c\") " Oct 11 10:31:44.971637 master-1 kubenswrapper[4771]: I1011 10:31:44.971462 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "e2d8f859-38d1-4916-8262-ff865eb9982c" (UID: "e2d8f859-38d1-4916-8262-ff865eb9982c"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:44.971732 master-1 kubenswrapper[4771]: I1011 10:31:44.971674 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:44.971767 master-1 kubenswrapper[4771]: I1011 10:31:44.971733 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-var-lock" (OuterVolumeSpecName: "var-lock") pod "e2d8f859-38d1-4916-8262-ff865eb9982c" (UID: "e2d8f859-38d1-4916-8262-ff865eb9982c"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:31:44.976477 master-1 kubenswrapper[4771]: I1011 10:31:44.974522 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d8f859-38d1-4916-8262-ff865eb9982c-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "e2d8f859-38d1-4916-8262-ff865eb9982c" (UID: "e2d8f859-38d1-4916-8262-ff865eb9982c"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:31:45.072286 master-1 kubenswrapper[4771]: I1011 10:31:45.072226 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/e2d8f859-38d1-4916-8262-ff865eb9982c-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:45.072286 master-1 kubenswrapper[4771]: I1011 10:31:45.072255 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/e2d8f859-38d1-4916-8262-ff865eb9982c-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:31:45.533166 master-1 kubenswrapper[4771]: I1011 10:31:45.533105 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-4-master-1" event={"ID":"e2d8f859-38d1-4916-8262-ff865eb9982c","Type":"ContainerDied","Data":"8a6f9bd6405227d24a21db358afae4da6b7dc87d9e61c80b59ee89918fb69c92"} Oct 11 10:31:45.533166 master-1 kubenswrapper[4771]: I1011 10:31:45.533172 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a6f9bd6405227d24a21db358afae4da6b7dc87d9e61c80b59ee89918fb69c92" Oct 11 10:31:45.533751 master-1 kubenswrapper[4771]: I1011 10:31:45.533331 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-4-master-1" Oct 11 10:31:45.736097 master-1 kubenswrapper[4771]: I1011 10:31:45.736009 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-apiserver/apiserver-777cc846dc-qpmws"] Oct 11 10:31:45.736445 master-1 kubenswrapper[4771]: E1011 10:31:45.736347 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e2d8f859-38d1-4916-8262-ff865eb9982c" containerName="installer" Oct 11 10:31:45.736445 master-1 kubenswrapper[4771]: I1011 10:31:45.736415 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d8f859-38d1-4916-8262-ff865eb9982c" containerName="installer" Oct 11 10:31:45.736627 master-1 kubenswrapper[4771]: I1011 10:31:45.736600 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e2d8f859-38d1-4916-8262-ff865eb9982c" containerName="installer" Oct 11 10:31:45.737966 master-1 kubenswrapper[4771]: I1011 10:31:45.737909 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.741275 master-1 kubenswrapper[4771]: I1011 10:31:45.741220 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"etcd-serving-ca" Oct 11 10:31:45.741275 master-1 kubenswrapper[4771]: I1011 10:31:45.741234 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"etcd-client" Oct 11 10:31:45.741563 master-1 kubenswrapper[4771]: I1011 10:31:45.741388 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"kube-root-ca.crt" Oct 11 10:31:45.741563 master-1 kubenswrapper[4771]: I1011 10:31:45.741220 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"encryption-config-1" Oct 11 10:31:45.742273 master-1 kubenswrapper[4771]: I1011 10:31:45.742232 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"image-import-ca" Oct 11 10:31:45.742433 master-1 kubenswrapper[4771]: I1011 10:31:45.742296 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"openshift-service-ca.crt" Oct 11 10:31:45.742433 master-1 kubenswrapper[4771]: I1011 10:31:45.742415 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"serving-cert" Oct 11 10:31:45.743227 master-1 kubenswrapper[4771]: I1011 10:31:45.743164 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"config" Oct 11 10:31:45.743403 master-1 kubenswrapper[4771]: I1011 10:31:45.743318 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"audit-1" Oct 11 10:31:45.749098 master-1 kubenswrapper[4771]: I1011 10:31:45.749050 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-apiserver/apiserver-777cc846dc-qpmws"] Oct 11 10:31:45.752643 master-1 kubenswrapper[4771]: I1011 10:31:45.752596 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-apiserver"/"trusted-ca-bundle" Oct 11 10:31:45.780383 master-1 kubenswrapper[4771]: I1011 10:31:45.780093 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-node-pullsecrets\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.780515 master-1 kubenswrapper[4771]: I1011 10:31:45.780411 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.780515 master-1 kubenswrapper[4771]: I1011 10:31:45.780501 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-config\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.780681 master-1 kubenswrapper[4771]: I1011 10:31:45.780606 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdjh\" (UniqueName: \"kubernetes.io/projected/e2fb9636-0787-426e-bd5e-cba0ea823b2b-kube-api-access-dpdjh\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.780761 master-1 kubenswrapper[4771]: I1011 10:31:45.780693 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-serving-ca\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.780828 master-1 kubenswrapper[4771]: I1011 10:31:45.780785 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit-dir\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.780892 master-1 kubenswrapper[4771]: I1011 10:31:45.780839 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-encryption-config\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.781146 master-1 kubenswrapper[4771]: I1011 10:31:45.780866 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-serving-cert\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.781228 master-1 kubenswrapper[4771]: I1011 10:31:45.781210 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-client\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.781294 master-1 kubenswrapper[4771]: I1011 10:31:45.781255 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-trusted-ca-bundle\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.781384 master-1 kubenswrapper[4771]: I1011 10:31:45.781370 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-image-import-ca\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.882810 master-1 kubenswrapper[4771]: I1011 10:31:45.882687 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-client\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.882810 master-1 kubenswrapper[4771]: I1011 10:31:45.882795 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-trusted-ca-bundle\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.882870 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-image-import-ca\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.882933 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-node-pullsecrets\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.882975 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.883024 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-config\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.883087 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdjh\" (UniqueName: \"kubernetes.io/projected/e2fb9636-0787-426e-bd5e-cba0ea823b2b-kube-api-access-dpdjh\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.883121 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-serving-ca\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.883206 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit-dir\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.883212 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-node-pullsecrets\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.883453 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-encryption-config\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.883488 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-serving-cert\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.883872 master-1 kubenswrapper[4771]: I1011 10:31:45.883528 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit-dir\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.885655 master-1 kubenswrapper[4771]: I1011 10:31:45.884646 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.885655 master-1 kubenswrapper[4771]: I1011 10:31:45.884833 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-trusted-ca-bundle\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.885655 master-1 kubenswrapper[4771]: I1011 10:31:45.885424 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-config\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.885655 master-1 kubenswrapper[4771]: I1011 10:31:45.885478 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-image-import-ca\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.885956 master-1 kubenswrapper[4771]: I1011 10:31:45.885677 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-serving-ca\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.887125 master-1 kubenswrapper[4771]: I1011 10:31:45.887066 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-client\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.887471 master-1 kubenswrapper[4771]: I1011 10:31:45.887425 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-encryption-config\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.888898 master-1 kubenswrapper[4771]: I1011 10:31:45.888846 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-serving-cert\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:45.912670 master-1 kubenswrapper[4771]: I1011 10:31:45.912594 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdjh\" (UniqueName: \"kubernetes.io/projected/e2fb9636-0787-426e-bd5e-cba0ea823b2b-kube-api-access-dpdjh\") pod \"apiserver-777cc846dc-qpmws\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:46.057994 master-1 kubenswrapper[4771]: I1011 10:31:46.057895 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:46.544177 master-1 kubenswrapper[4771]: I1011 10:31:46.544097 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-apiserver/apiserver-777cc846dc-qpmws"] Oct 11 10:31:46.548665 master-1 kubenswrapper[4771]: I1011 10:31:46.547049 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0c6dd9eb5bc384e5fbc388e7a2f95c28","Type":"ContainerStarted","Data":"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5"} Oct 11 10:31:46.548665 master-1 kubenswrapper[4771]: I1011 10:31:46.547104 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0c6dd9eb5bc384e5fbc388e7a2f95c28","Type":"ContainerStarted","Data":"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb"} Oct 11 10:31:46.607178 master-1 kubenswrapper[4771]: I1011 10:31:46.607124 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:31:46.607511 master-1 kubenswrapper[4771]: I1011 10:31:46.607468 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:31:46.957798 master-1 kubenswrapper[4771]: I1011 10:31:46.957714 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-guard-master-1"] Oct 11 10:31:46.958512 master-1 kubenswrapper[4771]: I1011 10:31:46.958484 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:31:46.961810 master-1 kubenswrapper[4771]: I1011 10:31:46.961748 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-controller-manager"/"kube-root-ca.crt" Oct 11 10:31:46.962621 master-1 kubenswrapper[4771]: I1011 10:31:46.962556 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-kube-controller-manager"/"openshift-service-ca.crt" Oct 11 10:31:46.979100 master-1 kubenswrapper[4771]: I1011 10:31:46.979051 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-guard-master-1"] Oct 11 10:31:46.999638 master-1 kubenswrapper[4771]: I1011 10:31:46.999570 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szlf6\" (UniqueName: \"kubernetes.io/projected/a706deec-9223-4663-9db5-71147d242c34-kube-api-access-szlf6\") pod \"kube-controller-manager-guard-master-1\" (UID: \"a706deec-9223-4663-9db5-71147d242c34\") " pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:31:47.101190 master-1 kubenswrapper[4771]: I1011 10:31:47.101109 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-szlf6\" (UniqueName: \"kubernetes.io/projected/a706deec-9223-4663-9db5-71147d242c34-kube-api-access-szlf6\") pod \"kube-controller-manager-guard-master-1\" (UID: \"a706deec-9223-4663-9db5-71147d242c34\") " pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:31:47.125920 master-1 kubenswrapper[4771]: I1011 10:31:47.125858 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-szlf6\" (UniqueName: \"kubernetes.io/projected/a706deec-9223-4663-9db5-71147d242c34-kube-api-access-szlf6\") pod \"kube-controller-manager-guard-master-1\" (UID: \"a706deec-9223-4663-9db5-71147d242c34\") " pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:31:47.292754 master-1 kubenswrapper[4771]: I1011 10:31:47.292631 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:31:47.545099 master-1 kubenswrapper[4771]: I1011 10:31:47.545019 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-guard-master-1"] Oct 11 10:31:47.548838 master-1 kubenswrapper[4771]: W1011 10:31:47.548779 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda706deec_9223_4663_9db5_71147d242c34.slice/crio-2a35a961efd9f1b6620917dd5d953d6d62a0fd18c86e7a1df3a9de3daf228dbf WatchSource:0}: Error finding container 2a35a961efd9f1b6620917dd5d953d6d62a0fd18c86e7a1df3a9de3daf228dbf: Status 404 returned error can't find the container with id 2a35a961efd9f1b6620917dd5d953d6d62a0fd18c86e7a1df3a9de3daf228dbf Oct 11 10:31:47.557904 master-1 kubenswrapper[4771]: I1011 10:31:47.557837 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0c6dd9eb5bc384e5fbc388e7a2f95c28","Type":"ContainerStarted","Data":"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1"} Oct 11 10:31:47.560393 master-1 kubenswrapper[4771]: I1011 10:31:47.560300 4771 generic.go:334] "Generic (PLEG): container finished" podID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerID="ed4ea2c827d3365e80a136d8fc9c70fdea44747628fc9e1b440208d196a14d73" exitCode=0 Oct 11 10:31:47.560526 master-1 kubenswrapper[4771]: I1011 10:31:47.560392 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" event={"ID":"e2fb9636-0787-426e-bd5e-cba0ea823b2b","Type":"ContainerDied","Data":"ed4ea2c827d3365e80a136d8fc9c70fdea44747628fc9e1b440208d196a14d73"} Oct 11 10:31:47.560526 master-1 kubenswrapper[4771]: I1011 10:31:47.560428 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" event={"ID":"e2fb9636-0787-426e-bd5e-cba0ea823b2b","Type":"ContainerStarted","Data":"a41a821c8fbcdc8c024fe125a36dfc655949ba099ab1bab4420d6e97047ce118"} Oct 11 10:31:47.586525 master-1 kubenswrapper[4771]: I1011 10:31:47.586440 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podStartSLOduration=4.586410382 podStartE2EDuration="4.586410382s" podCreationTimestamp="2025-10-11 10:31:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:31:47.582592236 +0000 UTC m=+339.556818717" watchObservedRunningTime="2025-10-11 10:31:47.586410382 +0000 UTC m=+339.560636823" Oct 11 10:31:47.711477 master-1 kubenswrapper[4771]: I1011 10:31:47.711326 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/kube-apiserver-guard-master-1"] Oct 11 10:31:47.781003 master-1 kubenswrapper[4771]: I1011 10:31:47.780954 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:31:48.569599 master-1 kubenswrapper[4771]: I1011 10:31:48.569523 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" event={"ID":"e2fb9636-0787-426e-bd5e-cba0ea823b2b","Type":"ContainerStarted","Data":"9b7973318d321c4747b9166204be01b90470f6b7ff6c1031063eb5d24ec05b0e"} Oct 11 10:31:48.569599 master-1 kubenswrapper[4771]: I1011 10:31:48.569595 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" event={"ID":"e2fb9636-0787-426e-bd5e-cba0ea823b2b","Type":"ContainerStarted","Data":"5314d6ef2281ac080baefb268e1b24e3959c52d75eecf8bba9e60d0238801c00"} Oct 11 10:31:48.584167 master-1 kubenswrapper[4771]: I1011 10:31:48.584073 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" event={"ID":"a706deec-9223-4663-9db5-71147d242c34","Type":"ContainerStarted","Data":"1c31430bd5d9e081e88aabc1ad810a536394ce113461601cc517a81f452f8976"} Oct 11 10:31:48.584167 master-1 kubenswrapper[4771]: I1011 10:31:48.584167 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" event={"ID":"a706deec-9223-4663-9db5-71147d242c34","Type":"ContainerStarted","Data":"2a35a961efd9f1b6620917dd5d953d6d62a0fd18c86e7a1df3a9de3daf228dbf"} Oct 11 10:31:48.584514 master-1 kubenswrapper[4771]: I1011 10:31:48.584192 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:31:48.590146 master-1 kubenswrapper[4771]: I1011 10:31:48.590094 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:31:48.606143 master-1 kubenswrapper[4771]: I1011 10:31:48.606050 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podStartSLOduration=57.60603043 podStartE2EDuration="57.60603043s" podCreationTimestamp="2025-10-11 10:30:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:31:48.60304228 +0000 UTC m=+340.577268751" watchObservedRunningTime="2025-10-11 10:31:48.60603043 +0000 UTC m=+340.580256901" Oct 11 10:31:48.621221 master-1 kubenswrapper[4771]: I1011 10:31:48.621149 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podStartSLOduration=2.621136956 podStartE2EDuration="2.621136956s" podCreationTimestamp="2025-10-11 10:31:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:31:48.620386713 +0000 UTC m=+340.594613164" watchObservedRunningTime="2025-10-11 10:31:48.621136956 +0000 UTC m=+340.595363417" Oct 11 10:31:51.058919 master-1 kubenswrapper[4771]: I1011 10:31:51.058812 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:51.058919 master-1 kubenswrapper[4771]: I1011 10:31:51.058912 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:51.071342 master-1 kubenswrapper[4771]: I1011 10:31:51.071273 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:51.564308 master-1 kubenswrapper[4771]: I1011 10:31:51.564233 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-guard-master-1"] Oct 11 10:31:51.608077 master-1 kubenswrapper[4771]: I1011 10:31:51.607989 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:31:51.608077 master-1 kubenswrapper[4771]: I1011 10:31:51.608068 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:31:51.609824 master-1 kubenswrapper[4771]: I1011 10:31:51.609770 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:31:53.646043 master-1 kubenswrapper[4771]: I1011 10:31:53.645964 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:53.646043 master-1 kubenswrapper[4771]: I1011 10:31:53.646055 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:53.647019 master-1 kubenswrapper[4771]: I1011 10:31:53.646079 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:53.647019 master-1 kubenswrapper[4771]: I1011 10:31:53.646102 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:53.653236 master-1 kubenswrapper[4771]: I1011 10:31:53.653149 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:53.654696 master-1 kubenswrapper[4771]: I1011 10:31:53.654635 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:54.628960 master-1 kubenswrapper[4771]: I1011 10:31:54.628882 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:56.607910 master-1 kubenswrapper[4771]: I1011 10:31:56.607602 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:31:56.609130 master-1 kubenswrapper[4771]: I1011 10:31:56.607912 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:31:56.642395 master-1 kubenswrapper[4771]: I1011 10:31:56.642259 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:31:58.403523 master-1 kubenswrapper[4771]: I1011 10:31:58.403453 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:32:01.607940 master-1 kubenswrapper[4771]: I1011 10:32:01.607834 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:32:01.607940 master-1 kubenswrapper[4771]: I1011 10:32:01.607919 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:32:06.607598 master-1 kubenswrapper[4771]: I1011 10:32:06.607538 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:32:06.608168 master-1 kubenswrapper[4771]: I1011 10:32:06.607641 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:32:08.723791 master-1 kubenswrapper[4771]: I1011 10:32:08.723689 4771 generic.go:334] "Generic (PLEG): container finished" podID="a61df698d34d049669621b2249bfe758" containerID="0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577" exitCode=0 Oct 11 10:32:08.723791 master-1 kubenswrapper[4771]: I1011 10:32:08.723766 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"a61df698d34d049669621b2249bfe758","Type":"ContainerDied","Data":"0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577"} Oct 11 10:32:09.734887 master-1 kubenswrapper[4771]: I1011 10:32:09.734779 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"a61df698d34d049669621b2249bfe758","Type":"ContainerStarted","Data":"e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316"} Oct 11 10:32:09.734887 master-1 kubenswrapper[4771]: I1011 10:32:09.734854 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"a61df698d34d049669621b2249bfe758","Type":"ContainerStarted","Data":"e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5"} Oct 11 10:32:09.734887 master-1 kubenswrapper[4771]: I1011 10:32:09.734875 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"a61df698d34d049669621b2249bfe758","Type":"ContainerStarted","Data":"63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f"} Oct 11 10:32:09.735991 master-1 kubenswrapper[4771]: I1011 10:32:09.735171 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:32:09.756459 master-1 kubenswrapper[4771]: I1011 10:32:09.756371 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podStartSLOduration=31.756324966 podStartE2EDuration="31.756324966s" podCreationTimestamp="2025-10-11 10:31:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:32:09.754721978 +0000 UTC m=+361.728948429" watchObservedRunningTime="2025-10-11 10:32:09.756324966 +0000 UTC m=+361.730551427" Oct 11 10:32:11.613899 master-1 kubenswrapper[4771]: I1011 10:32:11.613819 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:32:16.782288 master-1 kubenswrapper[4771]: I1011 10:32:16.782191 4771 generic.go:334] "Generic (PLEG): container finished" podID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerID="d9d09acfb9b74efc71914e418c9f7ad84873a3a13515d6cfcddf159cfd555604" exitCode=0 Oct 11 10:32:16.783558 master-1 kubenswrapper[4771]: I1011 10:32:16.782342 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" event={"ID":"04cd4a19-2532-43d1-9144-1f59d9e52d19","Type":"ContainerDied","Data":"d9d09acfb9b74efc71914e418c9f7ad84873a3a13515d6cfcddf159cfd555604"} Oct 11 10:32:16.783859 master-1 kubenswrapper[4771]: I1011 10:32:16.783816 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" event={"ID":"04cd4a19-2532-43d1-9144-1f59d9e52d19","Type":"ContainerStarted","Data":"2fd6e0cb14ecdcadbf2571f6d4dd1d2a4a1e6cf999fc333d09b9fc98b284b780"} Oct 11 10:32:17.494341 master-1 kubenswrapper[4771]: I1011 10:32:17.494290 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:32:17.497547 master-1 kubenswrapper[4771]: I1011 10:32:17.497494 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:17.497547 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:17.497547 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:17.497547 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:17.497860 master-1 kubenswrapper[4771]: I1011 10:32:17.497594 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:17.797220 master-1 kubenswrapper[4771]: I1011 10:32:17.797042 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:32:18.497230 master-1 kubenswrapper[4771]: I1011 10:32:18.497147 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:18.497230 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:18.497230 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:18.497230 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:18.497657 master-1 kubenswrapper[4771]: I1011 10:32:18.497244 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:19.497808 master-1 kubenswrapper[4771]: I1011 10:32:19.497717 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:19.497808 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:19.497808 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:19.497808 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:19.498967 master-1 kubenswrapper[4771]: I1011 10:32:19.497833 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:20.497639 master-1 kubenswrapper[4771]: I1011 10:32:20.497539 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:20.497639 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:20.497639 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:20.497639 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:20.498450 master-1 kubenswrapper[4771]: I1011 10:32:20.497640 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:21.497656 master-1 kubenswrapper[4771]: I1011 10:32:21.497562 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:21.497656 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:21.497656 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:21.497656 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:21.498644 master-1 kubenswrapper[4771]: I1011 10:32:21.497660 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:22.498873 master-1 kubenswrapper[4771]: I1011 10:32:22.498773 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:22.498873 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:22.498873 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:22.498873 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:22.499817 master-1 kubenswrapper[4771]: I1011 10:32:22.498897 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:23.497985 master-1 kubenswrapper[4771]: I1011 10:32:23.497927 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:23.497985 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:23.497985 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:23.497985 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:23.498527 master-1 kubenswrapper[4771]: I1011 10:32:23.498018 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:24.497429 master-1 kubenswrapper[4771]: I1011 10:32:24.497327 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:24.497429 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:24.497429 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:24.497429 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:24.498054 master-1 kubenswrapper[4771]: I1011 10:32:24.497464 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:25.494307 master-1 kubenswrapper[4771]: I1011 10:32:25.494253 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:32:25.496941 master-1 kubenswrapper[4771]: I1011 10:32:25.496878 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:25.496941 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:25.496941 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:25.496941 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:25.497401 master-1 kubenswrapper[4771]: I1011 10:32:25.496959 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:26.498341 master-1 kubenswrapper[4771]: I1011 10:32:26.498215 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:26.498341 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:26.498341 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:26.498341 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:26.499316 master-1 kubenswrapper[4771]: I1011 10:32:26.498393 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:27.497278 master-1 kubenswrapper[4771]: I1011 10:32:27.497164 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:27.497278 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:27.497278 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:27.497278 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:27.497278 master-1 kubenswrapper[4771]: I1011 10:32:27.497240 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:28.498009 master-1 kubenswrapper[4771]: I1011 10:32:28.497880 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:28.498009 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:28.498009 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:28.498009 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:28.498009 master-1 kubenswrapper[4771]: I1011 10:32:28.497998 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:29.498592 master-1 kubenswrapper[4771]: I1011 10:32:29.498501 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:29.498592 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:29.498592 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:29.498592 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:29.499938 master-1 kubenswrapper[4771]: I1011 10:32:29.498635 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:30.496872 master-1 kubenswrapper[4771]: I1011 10:32:30.496826 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:30.496872 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:30.496872 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:30.496872 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:30.497262 master-1 kubenswrapper[4771]: I1011 10:32:30.497233 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:31.498123 master-1 kubenswrapper[4771]: I1011 10:32:31.498002 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:31.498123 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:31.498123 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:31.498123 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:31.498123 master-1 kubenswrapper[4771]: I1011 10:32:31.498103 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:32.497722 master-1 kubenswrapper[4771]: I1011 10:32:32.497517 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:32.497722 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:32.497722 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:32.497722 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:32.497722 master-1 kubenswrapper[4771]: I1011 10:32:32.497617 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:33.498055 master-1 kubenswrapper[4771]: I1011 10:32:33.497958 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:33.498055 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:33.498055 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:33.498055 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:33.498055 master-1 kubenswrapper[4771]: I1011 10:32:33.498063 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:34.497735 master-1 kubenswrapper[4771]: I1011 10:32:34.497580 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:34.497735 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:34.497735 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:34.497735 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:34.497735 master-1 kubenswrapper[4771]: I1011 10:32:34.497722 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:35.497700 master-1 kubenswrapper[4771]: I1011 10:32:35.497562 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:35.497700 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:35.497700 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:35.497700 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:35.497700 master-1 kubenswrapper[4771]: I1011 10:32:35.497669 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:36.497562 master-1 kubenswrapper[4771]: I1011 10:32:36.497413 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:36.497562 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:36.497562 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:36.497562 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:36.498547 master-1 kubenswrapper[4771]: I1011 10:32:36.497555 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:37.497199 master-1 kubenswrapper[4771]: I1011 10:32:37.497105 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:37.497199 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:37.497199 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:37.497199 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:37.497199 master-1 kubenswrapper[4771]: I1011 10:32:37.497188 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:38.497474 master-1 kubenswrapper[4771]: I1011 10:32:38.497328 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:38.497474 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:38.497474 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:38.497474 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:38.498882 master-1 kubenswrapper[4771]: I1011 10:32:38.497477 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:38.608514 master-1 kubenswrapper[4771]: I1011 10:32:38.608405 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk"] Oct 11 10:32:38.608961 master-1 kubenswrapper[4771]: I1011 10:32:38.608803 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" containerID="cri-o://79f8e8a3af9681261cf6c96297e08774526c159a1df96245fda7d956c1a72204" gracePeriod=120 Oct 11 10:32:39.497500 master-1 kubenswrapper[4771]: I1011 10:32:39.497427 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:39.497500 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:39.497500 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:39.497500 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:39.498568 master-1 kubenswrapper[4771]: I1011 10:32:39.497520 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:40.497412 master-1 kubenswrapper[4771]: I1011 10:32:40.497304 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:40.497412 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:40.497412 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:40.497412 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:40.498551 master-1 kubenswrapper[4771]: I1011 10:32:40.497420 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:40.911146 master-1 kubenswrapper[4771]: I1011 10:32:40.911025 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-etcd/installer-5-master-1"] Oct 11 10:32:40.912157 master-1 kubenswrapper[4771]: I1011 10:32:40.912087 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:40.968714 master-1 kubenswrapper[4771]: I1011 10:32:40.968650 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/installer-5-master-1"] Oct 11 10:32:40.969742 master-1 kubenswrapper[4771]: I1011 10:32:40.969692 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:40.969849 master-1 kubenswrapper[4771]: I1011 10:32:40.969767 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-var-lock\") pod \"installer-5-master-1\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:40.969919 master-1 kubenswrapper[4771]: I1011 10:32:40.969900 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kube-api-access\") pod \"installer-5-master-1\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: I1011 10:32:41.061902 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:32:41.061959 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:32:41.062808 master-1 kubenswrapper[4771]: I1011 10:32:41.061982 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:41.071136 master-1 kubenswrapper[4771]: I1011 10:32:41.071078 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kube-api-access\") pod \"installer-5-master-1\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:41.071240 master-1 kubenswrapper[4771]: I1011 10:32:41.071180 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:41.071288 master-1 kubenswrapper[4771]: I1011 10:32:41.071237 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-var-lock\") pod \"installer-5-master-1\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:41.071448 master-1 kubenswrapper[4771]: I1011 10:32:41.071423 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:41.071570 master-1 kubenswrapper[4771]: I1011 10:32:41.071450 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-var-lock\") pod \"installer-5-master-1\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:41.091552 master-1 kubenswrapper[4771]: I1011 10:32:41.091451 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kube-api-access\") pod \"installer-5-master-1\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:41.288640 master-1 kubenswrapper[4771]: I1011 10:32:41.288431 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-5-master-1" Oct 11 10:32:41.498113 master-1 kubenswrapper[4771]: I1011 10:32:41.497502 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:41.498113 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:41.498113 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:41.498113 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:41.498113 master-1 kubenswrapper[4771]: I1011 10:32:41.497645 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:41.773726 master-1 kubenswrapper[4771]: I1011 10:32:41.773591 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/installer-5-master-1"] Oct 11 10:32:41.957651 master-1 kubenswrapper[4771]: I1011 10:32:41.957596 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-5-master-1" event={"ID":"f0f830cc-d36c-4ccd-97cb-2d4a99726684","Type":"ContainerStarted","Data":"65e9818b973bf19dd26838510d379ddf1b30f23283f0995cf12628a1f6d4cb94"} Oct 11 10:32:42.497429 master-1 kubenswrapper[4771]: I1011 10:32:42.497170 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:42.497429 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:42.497429 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:42.497429 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:42.497429 master-1 kubenswrapper[4771]: I1011 10:32:42.497282 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:42.965830 master-1 kubenswrapper[4771]: I1011 10:32:42.965720 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-5-master-1" event={"ID":"f0f830cc-d36c-4ccd-97cb-2d4a99726684","Type":"ContainerStarted","Data":"2b7fb64c483453dbfbd93869288690ed38d6d29cb105ac6ec22c06d0d9551aa1"} Oct 11 10:32:42.986024 master-1 kubenswrapper[4771]: I1011 10:32:42.985909 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-etcd/installer-5-master-1" podStartSLOduration=2.985878544 podStartE2EDuration="2.985878544s" podCreationTimestamp="2025-10-11 10:32:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:32:42.981222409 +0000 UTC m=+394.955448860" watchObservedRunningTime="2025-10-11 10:32:42.985878544 +0000 UTC m=+394.960104985" Oct 11 10:32:43.497649 master-1 kubenswrapper[4771]: I1011 10:32:43.497549 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:43.497649 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:43.497649 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:43.497649 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:43.497649 master-1 kubenswrapper[4771]: I1011 10:32:43.497641 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:44.498032 master-1 kubenswrapper[4771]: I1011 10:32:44.497961 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:44.498032 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:44.498032 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:44.498032 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:44.499036 master-1 kubenswrapper[4771]: I1011 10:32:44.498645 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:45.496869 master-1 kubenswrapper[4771]: I1011 10:32:45.496780 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:45.496869 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:45.496869 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:45.496869 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:45.497492 master-1 kubenswrapper[4771]: I1011 10:32:45.496876 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: I1011 10:32:46.061190 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:32:46.061303 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:32:46.062098 master-1 kubenswrapper[4771]: I1011 10:32:46.061333 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:46.497810 master-1 kubenswrapper[4771]: I1011 10:32:46.497671 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:46.497810 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:46.497810 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:46.497810 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:46.497810 master-1 kubenswrapper[4771]: I1011 10:32:46.497790 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:47.497474 master-1 kubenswrapper[4771]: I1011 10:32:47.497388 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:47.497474 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:47.497474 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:47.497474 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:47.498462 master-1 kubenswrapper[4771]: I1011 10:32:47.497500 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:47.780153 master-1 kubenswrapper[4771]: I1011 10:32:47.780004 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:32:48.498053 master-1 kubenswrapper[4771]: I1011 10:32:48.497940 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:48.498053 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:48.498053 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:48.498053 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:48.498788 master-1 kubenswrapper[4771]: I1011 10:32:48.498106 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:49.496730 master-1 kubenswrapper[4771]: I1011 10:32:49.496681 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:49.496730 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:49.496730 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:49.496730 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:49.497029 master-1 kubenswrapper[4771]: I1011 10:32:49.496740 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:50.498272 master-1 kubenswrapper[4771]: I1011 10:32:50.498053 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:50.498272 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:50.498272 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:50.498272 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:50.499347 master-1 kubenswrapper[4771]: I1011 10:32:50.498273 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: I1011 10:32:51.063971 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:32:51.064078 master-1 kubenswrapper[4771]: I1011 10:32:51.064076 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:51.065117 master-1 kubenswrapper[4771]: I1011 10:32:51.064254 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:32:51.498097 master-1 kubenswrapper[4771]: I1011 10:32:51.497847 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:51.498097 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:51.498097 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:51.498097 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:51.498097 master-1 kubenswrapper[4771]: I1011 10:32:51.497953 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:52.097572 master-1 kubenswrapper[4771]: E1011 10:32:52.097432 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" podUID="d7647696-42d9-4dd9-bc3b-a4d52a42cf9a" Oct 11 10:32:52.098199 master-1 kubenswrapper[4771]: E1011 10:32:52.097973 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" podUID="6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b" Oct 11 10:32:52.498303 master-1 kubenswrapper[4771]: I1011 10:32:52.498092 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:52.498303 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:52.498303 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:52.498303 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:52.498303 master-1 kubenswrapper[4771]: I1011 10:32:52.498192 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:53.026865 master-1 kubenswrapper[4771]: I1011 10:32:53.026757 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:32:53.027234 master-1 kubenswrapper[4771]: I1011 10:32:53.026781 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:32:53.497177 master-1 kubenswrapper[4771]: I1011 10:32:53.497095 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:53.497177 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:53.497177 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:53.497177 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:53.497722 master-1 kubenswrapper[4771]: I1011 10:32:53.497193 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:54.497917 master-1 kubenswrapper[4771]: I1011 10:32:54.497853 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:54.497917 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:54.497917 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:54.497917 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:54.498766 master-1 kubenswrapper[4771]: I1011 10:32:54.497946 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:55.394382 master-1 kubenswrapper[4771]: I1011 10:32:55.394284 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:32:55.394654 master-1 kubenswrapper[4771]: E1011 10:32:55.394600 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:34:57.394559322 +0000 UTC m=+529.368785953 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:32:55.495685 master-1 kubenswrapper[4771]: I1011 10:32:55.495568 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:32:55.496024 master-1 kubenswrapper[4771]: E1011 10:32:55.495980 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:34:57.495932801 +0000 UTC m=+529.470159282 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:32:55.497886 master-1 kubenswrapper[4771]: I1011 10:32:55.497836 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:55.497886 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:55.497886 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:55.497886 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:55.498501 master-1 kubenswrapper[4771]: I1011 10:32:55.497907 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: I1011 10:32:56.062971 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:32:56.063108 master-1 kubenswrapper[4771]: I1011 10:32:56.063045 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:56.497936 master-1 kubenswrapper[4771]: I1011 10:32:56.497724 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:56.497936 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:56.497936 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:56.497936 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:56.497936 master-1 kubenswrapper[4771]: I1011 10:32:56.497817 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:57.498849 master-1 kubenswrapper[4771]: I1011 10:32:57.498742 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:57.498849 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:57.498849 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:57.498849 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:57.500218 master-1 kubenswrapper[4771]: I1011 10:32:57.498871 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:58.498019 master-1 kubenswrapper[4771]: I1011 10:32:58.497931 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:58.498019 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:58.498019 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:58.498019 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:58.498514 master-1 kubenswrapper[4771]: I1011 10:32:58.498031 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:32:58.536971 master-1 kubenswrapper[4771]: I1011 10:32:58.536885 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:32:59.147847 master-1 kubenswrapper[4771]: E1011 10:32:59.147719 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" podUID="537a2b50-0394-47bd-941a-def350316943" Oct 11 10:32:59.496971 master-1 kubenswrapper[4771]: I1011 10:32:59.496687 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:32:59.496971 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:32:59.496971 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:32:59.496971 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:32:59.496971 master-1 kubenswrapper[4771]: I1011 10:32:59.496836 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:00.067575 master-1 kubenswrapper[4771]: I1011 10:33:00.067493 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:33:00.497189 master-1 kubenswrapper[4771]: I1011 10:33:00.496992 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:00.497189 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:00.497189 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:00.497189 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:00.497189 master-1 kubenswrapper[4771]: I1011 10:33:00.497094 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: I1011 10:33:01.061858 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:33:01.061968 master-1 kubenswrapper[4771]: I1011 10:33:01.061960 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:01.159815 master-1 kubenswrapper[4771]: E1011 10:33:01.159635 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" podUID="c9e9455e-0b47-4623-9b4c-ef79cf62a254" Oct 11 10:33:01.497953 master-1 kubenswrapper[4771]: I1011 10:33:01.497772 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:01.497953 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:01.497953 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:01.497953 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:01.497953 master-1 kubenswrapper[4771]: I1011 10:33:01.497874 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:02.080628 master-1 kubenswrapper[4771]: I1011 10:33:02.080471 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:33:02.292902 master-1 kubenswrapper[4771]: I1011 10:33:02.292785 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:33:02.293800 master-1 kubenswrapper[4771]: E1011 10:33:02.292999 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:33:02.293800 master-1 kubenswrapper[4771]: E1011 10:33:02.293187 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:35:04.293145902 +0000 UTC m=+536.267372513 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:33:02.498172 master-1 kubenswrapper[4771]: I1011 10:33:02.497934 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:02.498172 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:02.498172 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:02.498172 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:02.498172 master-1 kubenswrapper[4771]: I1011 10:33:02.498016 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:03.497501 master-1 kubenswrapper[4771]: I1011 10:33:03.497431 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:03.497501 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:03.497501 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:03.497501 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:03.498822 master-1 kubenswrapper[4771]: I1011 10:33:03.498693 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:04.498239 master-1 kubenswrapper[4771]: I1011 10:33:04.498116 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:04.498239 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:04.498239 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:04.498239 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:04.498239 master-1 kubenswrapper[4771]: I1011 10:33:04.498234 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:04.722630 master-1 kubenswrapper[4771]: I1011 10:33:04.722498 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:33:04.723100 master-1 kubenswrapper[4771]: E1011 10:33:04.722743 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:33:04.723100 master-1 kubenswrapper[4771]: E1011 10:33:04.722907 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:35:06.722869174 +0000 UTC m=+538.697095645 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:33:05.499040 master-1 kubenswrapper[4771]: I1011 10:33:05.498889 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:05.499040 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:05.499040 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:05.499040 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:05.499040 master-1 kubenswrapper[4771]: I1011 10:33:05.499021 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: I1011 10:33:06.062811 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:33:06.062937 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:33:06.063811 master-1 kubenswrapper[4771]: I1011 10:33:06.062962 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:06.497525 master-1 kubenswrapper[4771]: I1011 10:33:06.497378 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:06.497525 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:06.497525 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:06.497525 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:06.497525 master-1 kubenswrapper[4771]: I1011 10:33:06.497454 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:07.497565 master-1 kubenswrapper[4771]: I1011 10:33:07.497482 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:07.497565 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:07.497565 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:07.497565 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:07.498521 master-1 kubenswrapper[4771]: I1011 10:33:07.497580 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:08.498236 master-1 kubenswrapper[4771]: I1011 10:33:08.498151 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:08.498236 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:08.498236 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:08.498236 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:08.499175 master-1 kubenswrapper[4771]: I1011 10:33:08.498254 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:09.497000 master-1 kubenswrapper[4771]: I1011 10:33:09.496877 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:09.497000 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:09.497000 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:09.497000 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:09.497000 master-1 kubenswrapper[4771]: I1011 10:33:09.496966 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:10.497298 master-1 kubenswrapper[4771]: I1011 10:33:10.497194 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:10.497298 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:10.497298 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:10.497298 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:10.497298 master-1 kubenswrapper[4771]: I1011 10:33:10.497287 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: I1011 10:33:11.063677 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:33:11.063848 master-1 kubenswrapper[4771]: I1011 10:33:11.063780 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:11.497162 master-1 kubenswrapper[4771]: I1011 10:33:11.497026 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:11.497162 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:11.497162 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:11.497162 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:11.497912 master-1 kubenswrapper[4771]: I1011 10:33:11.497879 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:12.421590 master-1 kubenswrapper[4771]: I1011 10:33:12.421491 4771 scope.go:117] "RemoveContainer" containerID="a31d75d150e0d2dcf8878fd1b60bee95ea19d0157365ef6735168ff809442b4b" Oct 11 10:33:12.497235 master-1 kubenswrapper[4771]: I1011 10:33:12.497155 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:12.497235 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:12.497235 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:12.497235 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:12.498121 master-1 kubenswrapper[4771]: I1011 10:33:12.497265 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:13.497002 master-1 kubenswrapper[4771]: I1011 10:33:13.496876 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:13.497002 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:13.497002 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:13.497002 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:13.497002 master-1 kubenswrapper[4771]: I1011 10:33:13.496952 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:13.726937 master-1 kubenswrapper[4771]: I1011 10:33:13.726869 4771 kubelet.go:2431] "SyncLoop REMOVE" source="file" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:33:13.728306 master-1 kubenswrapper[4771]: I1011 10:33:13.727901 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcdctl" containerID="cri-o://2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8" gracePeriod=30 Oct 11 10:33:13.728726 master-1 kubenswrapper[4771]: I1011 10:33:13.727965 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" containerID="cri-o://8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e" gracePeriod=30 Oct 11 10:33:13.728883 master-1 kubenswrapper[4771]: I1011 10:33:13.727966 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-readyz" containerID="cri-o://cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711" gracePeriod=30 Oct 11 10:33:13.728883 master-1 kubenswrapper[4771]: I1011 10:33:13.728058 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-rev" containerID="cri-o://958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0" gracePeriod=30 Oct 11 10:33:13.728883 master-1 kubenswrapper[4771]: I1011 10:33:13.728108 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-metrics" containerID="cri-o://4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0" gracePeriod=30 Oct 11 10:33:13.731874 master-1 kubenswrapper[4771]: I1011 10:33:13.731699 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:33:13.732045 master-1 kubenswrapper[4771]: E1011 10:33:13.731934 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" Oct 11 10:33:13.732045 master-1 kubenswrapper[4771]: I1011 10:33:13.731952 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" Oct 11 10:33:13.732045 master-1 kubenswrapper[4771]: E1011 10:33:13.731971 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-ensure-env-vars" Oct 11 10:33:13.732045 master-1 kubenswrapper[4771]: I1011 10:33:13.731981 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-ensure-env-vars" Oct 11 10:33:13.732045 master-1 kubenswrapper[4771]: E1011 10:33:13.731998 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-readyz" Oct 11 10:33:13.732045 master-1 kubenswrapper[4771]: I1011 10:33:13.732008 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-readyz" Oct 11 10:33:13.732045 master-1 kubenswrapper[4771]: E1011 10:33:13.732023 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-rev" Oct 11 10:33:13.732045 master-1 kubenswrapper[4771]: I1011 10:33:13.732032 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-rev" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: E1011 10:33:13.732119 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="setup" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732133 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="setup" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: E1011 10:33:13.732144 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcdctl" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732153 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcdctl" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: E1011 10:33:13.732164 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732173 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: E1011 10:33:13.732184 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-metrics" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732193 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-metrics" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: E1011 10:33:13.732204 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-resources-copy" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732214 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-resources-copy" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732352 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-rev" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732377 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732409 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-metrics" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732419 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcdctl" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732433 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732447 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd-readyz" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732461 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: E1011 10:33:13.732587 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" Oct 11 10:33:13.732861 master-1 kubenswrapper[4771]: I1011 10:33:13.732600 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerName="etcd" Oct 11 10:33:13.752643 master-1 kubenswrapper[4771]: I1011 10:33:13.752531 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-log-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.752643 master-1 kubenswrapper[4771]: I1011 10:33:13.752609 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-static-pod-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.753214 master-1 kubenswrapper[4771]: I1011 10:33:13.752673 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-usr-local-bin\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.753214 master-1 kubenswrapper[4771]: I1011 10:33:13.752731 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-cert-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.753214 master-1 kubenswrapper[4771]: I1011 10:33:13.752764 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-resource-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.753214 master-1 kubenswrapper[4771]: I1011 10:33:13.752811 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-data-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854555 master-1 kubenswrapper[4771]: I1011 10:33:13.854486 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-log-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854555 master-1 kubenswrapper[4771]: I1011 10:33:13.854546 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-static-pod-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854575 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-usr-local-bin\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854620 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-cert-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854641 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-resource-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854645 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-log-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854745 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-usr-local-bin\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854773 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-data-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854679 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-data-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854812 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-resource-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854749 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-static-pod-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:13.854856 master-1 kubenswrapper[4771]: I1011 10:33:13.854820 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-cert-dir\") pod \"etcd-master-1\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:33:14.160397 master-1 kubenswrapper[4771]: I1011 10:33:14.160295 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/1.log" Oct 11 10:33:14.161203 master-1 kubenswrapper[4771]: I1011 10:33:14.161137 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-rev/0.log" Oct 11 10:33:14.162750 master-1 kubenswrapper[4771]: I1011 10:33:14.162695 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-metrics/0.log" Oct 11 10:33:14.165068 master-1 kubenswrapper[4771]: I1011 10:33:14.164991 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0" exitCode=2 Oct 11 10:33:14.165068 master-1 kubenswrapper[4771]: I1011 10:33:14.165036 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711" exitCode=0 Oct 11 10:33:14.165068 master-1 kubenswrapper[4771]: I1011 10:33:14.165052 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0" exitCode=2 Oct 11 10:33:14.167165 master-1 kubenswrapper[4771]: I1011 10:33:14.167097 4771 generic.go:334] "Generic (PLEG): container finished" podID="f0f830cc-d36c-4ccd-97cb-2d4a99726684" containerID="2b7fb64c483453dbfbd93869288690ed38d6d29cb105ac6ec22c06d0d9551aa1" exitCode=0 Oct 11 10:33:14.167165 master-1 kubenswrapper[4771]: I1011 10:33:14.167140 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-5-master-1" event={"ID":"f0f830cc-d36c-4ccd-97cb-2d4a99726684","Type":"ContainerDied","Data":"2b7fb64c483453dbfbd93869288690ed38d6d29cb105ac6ec22c06d0d9551aa1"} Oct 11 10:33:14.173801 master-1 kubenswrapper[4771]: I1011 10:33:14.173725 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-master-1" oldPodUID="5268b2f2ae2aef0c7f2e7a6e651ed702" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" Oct 11 10:33:14.497719 master-1 kubenswrapper[4771]: I1011 10:33:14.497552 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:14.497719 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:14.497719 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:14.497719 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:14.497719 master-1 kubenswrapper[4771]: I1011 10:33:14.497648 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:14.629223 master-1 kubenswrapper[4771]: I1011 10:33:14.629117 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:14.629223 master-1 kubenswrapper[4771]: I1011 10:33:14.629217 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:15.502890 master-1 kubenswrapper[4771]: I1011 10:33:15.502812 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:15.502890 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:15.502890 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:15.502890 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:15.504223 master-1 kubenswrapper[4771]: I1011 10:33:15.502913 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:15.535152 master-1 kubenswrapper[4771]: I1011 10:33:15.535067 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-5-master-1" Oct 11 10:33:15.575033 master-1 kubenswrapper[4771]: I1011 10:33:15.574900 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kube-api-access\") pod \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " Oct 11 10:33:15.575311 master-1 kubenswrapper[4771]: I1011 10:33:15.575202 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kubelet-dir\") pod \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " Oct 11 10:33:15.575311 master-1 kubenswrapper[4771]: I1011 10:33:15.575295 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-var-lock\") pod \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\" (UID: \"f0f830cc-d36c-4ccd-97cb-2d4a99726684\") " Oct 11 10:33:15.575798 master-1 kubenswrapper[4771]: I1011 10:33:15.575648 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "f0f830cc-d36c-4ccd-97cb-2d4a99726684" (UID: "f0f830cc-d36c-4ccd-97cb-2d4a99726684"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:33:15.575798 master-1 kubenswrapper[4771]: I1011 10:33:15.575746 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-var-lock" (OuterVolumeSpecName: "var-lock") pod "f0f830cc-d36c-4ccd-97cb-2d4a99726684" (UID: "f0f830cc-d36c-4ccd-97cb-2d4a99726684"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:33:15.576348 master-1 kubenswrapper[4771]: I1011 10:33:15.576316 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:15.577850 master-1 kubenswrapper[4771]: I1011 10:33:15.576349 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/f0f830cc-d36c-4ccd-97cb-2d4a99726684-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:15.578643 master-1 kubenswrapper[4771]: I1011 10:33:15.578579 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "f0f830cc-d36c-4ccd-97cb-2d4a99726684" (UID: "f0f830cc-d36c-4ccd-97cb-2d4a99726684"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:33:15.677586 master-1 kubenswrapper[4771]: I1011 10:33:15.677483 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/f0f830cc-d36c-4ccd-97cb-2d4a99726684-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: I1011 10:33:16.064261 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:33:16.064342 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:33:16.065418 master-1 kubenswrapper[4771]: I1011 10:33:16.064402 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:16.185035 master-1 kubenswrapper[4771]: I1011 10:33:16.184955 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-5-master-1" event={"ID":"f0f830cc-d36c-4ccd-97cb-2d4a99726684","Type":"ContainerDied","Data":"65e9818b973bf19dd26838510d379ddf1b30f23283f0995cf12628a1f6d4cb94"} Oct 11 10:33:16.185035 master-1 kubenswrapper[4771]: I1011 10:33:16.185018 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e9818b973bf19dd26838510d379ddf1b30f23283f0995cf12628a1f6d4cb94" Oct 11 10:33:16.185434 master-1 kubenswrapper[4771]: I1011 10:33:16.185040 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-5-master-1" Oct 11 10:33:16.497479 master-1 kubenswrapper[4771]: I1011 10:33:16.497196 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:16.497479 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:16.497479 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:16.497479 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:16.497479 master-1 kubenswrapper[4771]: I1011 10:33:16.497300 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:17.497310 master-1 kubenswrapper[4771]: I1011 10:33:17.497222 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:17.497310 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:17.497310 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:17.497310 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:17.498023 master-1 kubenswrapper[4771]: I1011 10:33:17.497324 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:17.791087 master-1 kubenswrapper[4771]: I1011 10:33:17.790932 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:33:18.497622 master-1 kubenswrapper[4771]: I1011 10:33:18.497505 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:18.497622 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:18.497622 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:18.497622 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:18.497622 master-1 kubenswrapper[4771]: I1011 10:33:18.497604 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:19.497292 master-1 kubenswrapper[4771]: I1011 10:33:19.497219 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:19.497292 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:19.497292 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:19.497292 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:19.497615 master-1 kubenswrapper[4771]: I1011 10:33:19.497311 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:19.628440 master-1 kubenswrapper[4771]: I1011 10:33:19.628325 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:19.629187 master-1 kubenswrapper[4771]: I1011 10:33:19.628451 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:20.497874 master-1 kubenswrapper[4771]: I1011 10:33:20.497790 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:20.497874 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:20.497874 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:20.497874 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:20.497874 master-1 kubenswrapper[4771]: I1011 10:33:20.497871 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: I1011 10:33:21.065392 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:33:21.065462 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:33:21.066471 master-1 kubenswrapper[4771]: I1011 10:33:21.065498 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:21.497910 master-1 kubenswrapper[4771]: I1011 10:33:21.497767 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:21.497910 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:21.497910 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:21.497910 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:21.497910 master-1 kubenswrapper[4771]: I1011 10:33:21.497863 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:22.497334 master-1 kubenswrapper[4771]: I1011 10:33:22.497243 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:22.497334 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:22.497334 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:22.497334 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:22.498272 master-1 kubenswrapper[4771]: I1011 10:33:22.497352 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:23.500885 master-1 kubenswrapper[4771]: I1011 10:33:23.500752 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:23.500885 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:23.500885 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:23.500885 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:23.501855 master-1 kubenswrapper[4771]: I1011 10:33:23.500907 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:24.498449 master-1 kubenswrapper[4771]: I1011 10:33:24.498344 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:24.498449 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:24.498449 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:24.498449 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:24.498449 master-1 kubenswrapper[4771]: I1011 10:33:24.498447 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:24.628577 master-1 kubenswrapper[4771]: I1011 10:33:24.628472 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:24.629596 master-1 kubenswrapper[4771]: I1011 10:33:24.628577 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:24.629596 master-1 kubenswrapper[4771]: I1011 10:33:24.628715 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:33:24.629821 master-1 kubenswrapper[4771]: I1011 10:33:24.629622 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:24.629821 master-1 kubenswrapper[4771]: I1011 10:33:24.629718 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:25.497312 master-1 kubenswrapper[4771]: I1011 10:33:25.497130 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:25.497312 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:25.497312 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:25.497312 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:25.497312 master-1 kubenswrapper[4771]: I1011 10:33:25.497327 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: I1011 10:33:26.065869 4771 patch_prober.go:28] interesting pod/apiserver-6f855d6bcf-cwmmk container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:33:26.066023 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:33:26.067599 master-1 kubenswrapper[4771]: I1011 10:33:26.066055 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:26.497527 master-1 kubenswrapper[4771]: I1011 10:33:26.497317 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:26.497527 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:26.497527 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:26.497527 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:26.497527 master-1 kubenswrapper[4771]: I1011 10:33:26.497440 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:27.497318 master-1 kubenswrapper[4771]: I1011 10:33:27.497209 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:27.497318 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:27.497318 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:27.497318 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:27.497318 master-1 kubenswrapper[4771]: I1011 10:33:27.497292 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:28.498246 master-1 kubenswrapper[4771]: I1011 10:33:28.498129 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:28.498246 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:28.498246 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:28.498246 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:28.498246 master-1 kubenswrapper[4771]: I1011 10:33:28.498260 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:29.496611 master-1 kubenswrapper[4771]: I1011 10:33:29.496529 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:29.496611 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:29.496611 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:29.496611 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:29.497052 master-1 kubenswrapper[4771]: I1011 10:33:29.496689 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:29.628323 master-1 kubenswrapper[4771]: I1011 10:33:29.628244 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:29.628323 master-1 kubenswrapper[4771]: I1011 10:33:29.628328 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:30.278441 master-1 kubenswrapper[4771]: I1011 10:33:30.277430 4771 generic.go:334] "Generic (PLEG): container finished" podID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerID="79f8e8a3af9681261cf6c96297e08774526c159a1df96245fda7d956c1a72204" exitCode=0 Oct 11 10:33:30.278441 master-1 kubenswrapper[4771]: I1011 10:33:30.277506 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" event={"ID":"d87cc032-b419-444c-8bf0-ef7405d7369d","Type":"ContainerDied","Data":"79f8e8a3af9681261cf6c96297e08774526c159a1df96245fda7d956c1a72204"} Oct 11 10:33:30.498256 master-1 kubenswrapper[4771]: I1011 10:33:30.497600 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:30.498256 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:30.498256 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:30.498256 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:30.498256 master-1 kubenswrapper[4771]: I1011 10:33:30.497691 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:30.640493 master-1 kubenswrapper[4771]: I1011 10:33:30.640400 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:33:30.689533 master-1 kubenswrapper[4771]: I1011 10:33:30.689435 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-trusted-ca-bundle\") pod \"d87cc032-b419-444c-8bf0-ef7405d7369d\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " Oct 11 10:33:30.689821 master-1 kubenswrapper[4771]: I1011 10:33:30.689577 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-policies\") pod \"d87cc032-b419-444c-8bf0-ef7405d7369d\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " Oct 11 10:33:30.689821 master-1 kubenswrapper[4771]: I1011 10:33:30.689652 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-serving-ca\") pod \"d87cc032-b419-444c-8bf0-ef7405d7369d\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " Oct 11 10:33:30.689821 master-1 kubenswrapper[4771]: I1011 10:33:30.689689 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kldk2\" (UniqueName: \"kubernetes.io/projected/d87cc032-b419-444c-8bf0-ef7405d7369d-kube-api-access-kldk2\") pod \"d87cc032-b419-444c-8bf0-ef7405d7369d\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " Oct 11 10:33:30.689821 master-1 kubenswrapper[4771]: I1011 10:33:30.689733 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-encryption-config\") pod \"d87cc032-b419-444c-8bf0-ef7405d7369d\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " Oct 11 10:33:30.689821 master-1 kubenswrapper[4771]: I1011 10:33:30.689780 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-serving-cert\") pod \"d87cc032-b419-444c-8bf0-ef7405d7369d\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " Oct 11 10:33:30.689821 master-1 kubenswrapper[4771]: I1011 10:33:30.689812 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-dir\") pod \"d87cc032-b419-444c-8bf0-ef7405d7369d\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " Oct 11 10:33:30.690323 master-1 kubenswrapper[4771]: I1011 10:33:30.689848 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-client\") pod \"d87cc032-b419-444c-8bf0-ef7405d7369d\" (UID: \"d87cc032-b419-444c-8bf0-ef7405d7369d\") " Oct 11 10:33:30.690323 master-1 kubenswrapper[4771]: I1011 10:33:30.690012 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "d87cc032-b419-444c-8bf0-ef7405d7369d" (UID: "d87cc032-b419-444c-8bf0-ef7405d7369d"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:33:30.690323 master-1 kubenswrapper[4771]: I1011 10:33:30.690187 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:30.690961 master-1 kubenswrapper[4771]: I1011 10:33:30.690860 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-policies" (OuterVolumeSpecName: "audit-policies") pod "d87cc032-b419-444c-8bf0-ef7405d7369d" (UID: "d87cc032-b419-444c-8bf0-ef7405d7369d"). InnerVolumeSpecName "audit-policies". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:33:30.691109 master-1 kubenswrapper[4771]: I1011 10:33:30.691004 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-serving-ca" (OuterVolumeSpecName: "etcd-serving-ca") pod "d87cc032-b419-444c-8bf0-ef7405d7369d" (UID: "d87cc032-b419-444c-8bf0-ef7405d7369d"). InnerVolumeSpecName "etcd-serving-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:33:30.691403 master-1 kubenswrapper[4771]: I1011 10:33:30.691298 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d87cc032-b419-444c-8bf0-ef7405d7369d" (UID: "d87cc032-b419-444c-8bf0-ef7405d7369d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:33:30.694160 master-1 kubenswrapper[4771]: I1011 10:33:30.694097 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87cc032-b419-444c-8bf0-ef7405d7369d-kube-api-access-kldk2" (OuterVolumeSpecName: "kube-api-access-kldk2") pod "d87cc032-b419-444c-8bf0-ef7405d7369d" (UID: "d87cc032-b419-444c-8bf0-ef7405d7369d"). InnerVolumeSpecName "kube-api-access-kldk2". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:33:30.694330 master-1 kubenswrapper[4771]: I1011 10:33:30.694289 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-client" (OuterVolumeSpecName: "etcd-client") pod "d87cc032-b419-444c-8bf0-ef7405d7369d" (UID: "d87cc032-b419-444c-8bf0-ef7405d7369d"). InnerVolumeSpecName "etcd-client". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:33:30.694858 master-1 kubenswrapper[4771]: I1011 10:33:30.694787 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-encryption-config" (OuterVolumeSpecName: "encryption-config") pod "d87cc032-b419-444c-8bf0-ef7405d7369d" (UID: "d87cc032-b419-444c-8bf0-ef7405d7369d"). InnerVolumeSpecName "encryption-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:33:30.695109 master-1 kubenswrapper[4771]: I1011 10:33:30.695043 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "d87cc032-b419-444c-8bf0-ef7405d7369d" (UID: "d87cc032-b419-444c-8bf0-ef7405d7369d"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:33:30.792203 master-1 kubenswrapper[4771]: I1011 10:33:30.792043 4771 reconciler_common.go:293] "Volume detached for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-audit-policies\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:30.792203 master-1 kubenswrapper[4771]: I1011 10:33:30.792131 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-serving-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:30.792203 master-1 kubenswrapper[4771]: I1011 10:33:30.792155 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-kldk2\" (UniqueName: \"kubernetes.io/projected/d87cc032-b419-444c-8bf0-ef7405d7369d-kube-api-access-kldk2\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:30.792203 master-1 kubenswrapper[4771]: I1011 10:33:30.792178 4771 reconciler_common.go:293] "Volume detached for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-encryption-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:30.792203 master-1 kubenswrapper[4771]: I1011 10:33:30.792197 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:30.792203 master-1 kubenswrapper[4771]: I1011 10:33:30.792220 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/d87cc032-b419-444c-8bf0-ef7405d7369d-etcd-client\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:30.792203 master-1 kubenswrapper[4771]: I1011 10:33:30.792239 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d87cc032-b419-444c-8bf0-ef7405d7369d-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:31.286302 master-1 kubenswrapper[4771]: I1011 10:33:31.286221 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" event={"ID":"d87cc032-b419-444c-8bf0-ef7405d7369d","Type":"ContainerDied","Data":"f8786873b90c54bfb0b515ad88ba2ef097b9f25b5ded48493272a640d89c1d55"} Oct 11 10:33:31.286302 master-1 kubenswrapper[4771]: I1011 10:33:31.286289 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk" Oct 11 10:33:31.286888 master-1 kubenswrapper[4771]: I1011 10:33:31.286332 4771 scope.go:117] "RemoveContainer" containerID="79f8e8a3af9681261cf6c96297e08774526c159a1df96245fda7d956c1a72204" Oct 11 10:33:31.311497 master-1 kubenswrapper[4771]: I1011 10:33:31.311408 4771 scope.go:117] "RemoveContainer" containerID="cc3604bd3c6d5088cac6e57645a1372932a2b915b7df557349ccea609bf9af52" Oct 11 10:33:31.342322 master-1 kubenswrapper[4771]: I1011 10:33:31.342240 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk"] Oct 11 10:33:31.355621 master-1 kubenswrapper[4771]: I1011 10:33:31.355541 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-oauth-apiserver/apiserver-6f855d6bcf-cwmmk"] Oct 11 10:33:31.497927 master-1 kubenswrapper[4771]: I1011 10:33:31.497825 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:31.497927 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:31.497927 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:31.497927 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:31.497927 master-1 kubenswrapper[4771]: I1011 10:33:31.497925 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:32.447201 master-1 kubenswrapper[4771]: I1011 10:33:32.447084 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" path="/var/lib/kubelet/pods/d87cc032-b419-444c-8bf0-ef7405d7369d/volumes" Oct 11 10:33:32.496853 master-1 kubenswrapper[4771]: I1011 10:33:32.496787 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:32.496853 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:32.496853 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:32.496853 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:32.497166 master-1 kubenswrapper[4771]: I1011 10:33:32.496875 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:33.497842 master-1 kubenswrapper[4771]: I1011 10:33:33.497752 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:33.497842 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:33.497842 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:33.497842 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:33.498863 master-1 kubenswrapper[4771]: I1011 10:33:33.497862 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:34.497769 master-1 kubenswrapper[4771]: I1011 10:33:34.497654 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:34.497769 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:34.497769 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:34.497769 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:34.497769 master-1 kubenswrapper[4771]: I1011 10:33:34.497752 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:34.628916 master-1 kubenswrapper[4771]: I1011 10:33:34.628789 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:34.628916 master-1 kubenswrapper[4771]: I1011 10:33:34.628898 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:35.497332 master-1 kubenswrapper[4771]: I1011 10:33:35.497230 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:35.497332 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:35.497332 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:35.497332 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:35.497781 master-1 kubenswrapper[4771]: I1011 10:33:35.497341 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:35.807297 master-1 kubenswrapper[4771]: I1011 10:33:35.807172 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b"] Oct 11 10:33:35.808113 master-1 kubenswrapper[4771]: E1011 10:33:35.807480 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" Oct 11 10:33:35.808113 master-1 kubenswrapper[4771]: I1011 10:33:35.807502 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" Oct 11 10:33:35.808113 master-1 kubenswrapper[4771]: E1011 10:33:35.807529 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="f0f830cc-d36c-4ccd-97cb-2d4a99726684" containerName="installer" Oct 11 10:33:35.808113 master-1 kubenswrapper[4771]: I1011 10:33:35.807542 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f830cc-d36c-4ccd-97cb-2d4a99726684" containerName="installer" Oct 11 10:33:35.808113 master-1 kubenswrapper[4771]: E1011 10:33:35.807561 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="fix-audit-permissions" Oct 11 10:33:35.808113 master-1 kubenswrapper[4771]: I1011 10:33:35.807573 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="fix-audit-permissions" Oct 11 10:33:35.808113 master-1 kubenswrapper[4771]: I1011 10:33:35.807717 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="f0f830cc-d36c-4ccd-97cb-2d4a99726684" containerName="installer" Oct 11 10:33:35.808113 master-1 kubenswrapper[4771]: I1011 10:33:35.807739 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d87cc032-b419-444c-8bf0-ef7405d7369d" containerName="oauth-apiserver" Oct 11 10:33:35.808929 master-1 kubenswrapper[4771]: I1011 10:33:35.808906 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.812266 master-1 kubenswrapper[4771]: I1011 10:33:35.812174 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"encryption-config-1" Oct 11 10:33:35.813138 master-1 kubenswrapper[4771]: I1011 10:33:35.812687 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"openshift-service-ca.crt" Oct 11 10:33:35.813138 master-1 kubenswrapper[4771]: I1011 10:33:35.813118 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"trusted-ca-bundle" Oct 11 10:33:35.813438 master-1 kubenswrapper[4771]: I1011 10:33:35.813290 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"etcd-client" Oct 11 10:33:35.814312 master-1 kubenswrapper[4771]: I1011 10:33:35.814240 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"kube-root-ca.crt" Oct 11 10:33:35.814312 master-1 kubenswrapper[4771]: I1011 10:33:35.814288 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"etcd-serving-ca" Oct 11 10:33:35.814592 master-1 kubenswrapper[4771]: I1011 10:33:35.814240 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"audit-1" Oct 11 10:33:35.814592 master-1 kubenswrapper[4771]: I1011 10:33:35.814286 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"serving-cert" Oct 11 10:33:35.834989 master-1 kubenswrapper[4771]: I1011 10:33:35.834895 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b"] Oct 11 10:33:35.859553 master-1 kubenswrapper[4771]: I1011 10:33:35.859487 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdvt\" (UniqueName: \"kubernetes.io/projected/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-kube-api-access-crdvt\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.859716 master-1 kubenswrapper[4771]: I1011 10:33:35.859607 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-serving-ca\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.859716 master-1 kubenswrapper[4771]: I1011 10:33:35.859671 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-dir\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.859855 master-1 kubenswrapper[4771]: I1011 10:33:35.859703 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-encryption-config\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.859855 master-1 kubenswrapper[4771]: I1011 10:33:35.859776 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-serving-cert\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.859855 master-1 kubenswrapper[4771]: I1011 10:33:35.859848 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-client\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.860046 master-1 kubenswrapper[4771]: I1011 10:33:35.859873 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-policies\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.860046 master-1 kubenswrapper[4771]: I1011 10:33:35.859933 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-trusted-ca-bundle\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.960444 master-1 kubenswrapper[4771]: I1011 10:33:35.960346 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-dir\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.960444 master-1 kubenswrapper[4771]: I1011 10:33:35.960433 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-encryption-config\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.960865 master-1 kubenswrapper[4771]: I1011 10:33:35.960491 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-serving-cert\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.960865 master-1 kubenswrapper[4771]: I1011 10:33:35.960528 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-client\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.960865 master-1 kubenswrapper[4771]: I1011 10:33:35.960550 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-policies\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.960865 master-1 kubenswrapper[4771]: I1011 10:33:35.960576 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-trusted-ca-bundle\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.960865 master-1 kubenswrapper[4771]: I1011 10:33:35.960587 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-dir\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.960865 master-1 kubenswrapper[4771]: I1011 10:33:35.960611 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-crdvt\" (UniqueName: \"kubernetes.io/projected/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-kube-api-access-crdvt\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.960865 master-1 kubenswrapper[4771]: I1011 10:33:35.960786 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-serving-ca\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.962205 master-1 kubenswrapper[4771]: I1011 10:33:35.962095 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-policies\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.962604 master-1 kubenswrapper[4771]: I1011 10:33:35.962536 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-serving-ca\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.962700 master-1 kubenswrapper[4771]: I1011 10:33:35.962590 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-trusted-ca-bundle\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.966056 master-1 kubenswrapper[4771]: I1011 10:33:35.965966 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-client\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.967081 master-1 kubenswrapper[4771]: I1011 10:33:35.966797 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-serving-cert\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.967212 master-1 kubenswrapper[4771]: I1011 10:33:35.967131 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-encryption-config\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:35.999401 master-1 kubenswrapper[4771]: I1011 10:33:35.999285 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdvt\" (UniqueName: \"kubernetes.io/projected/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-kube-api-access-crdvt\") pod \"apiserver-68f4c55ff4-z898b\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:36.136510 master-1 kubenswrapper[4771]: I1011 10:33:36.136254 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:36.496547 master-1 kubenswrapper[4771]: I1011 10:33:36.496345 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:36.496547 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:36.496547 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:36.496547 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:36.496547 master-1 kubenswrapper[4771]: I1011 10:33:36.496488 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:36.642016 master-1 kubenswrapper[4771]: I1011 10:33:36.641947 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b"] Oct 11 10:33:37.331041 master-1 kubenswrapper[4771]: I1011 10:33:37.330949 4771 generic.go:334] "Generic (PLEG): container finished" podID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerID="099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61" exitCode=0 Oct 11 10:33:37.331041 master-1 kubenswrapper[4771]: I1011 10:33:37.331013 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" event={"ID":"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40","Type":"ContainerDied","Data":"099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61"} Oct 11 10:33:37.331041 master-1 kubenswrapper[4771]: I1011 10:33:37.331048 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" event={"ID":"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40","Type":"ContainerStarted","Data":"ef282371271fc7902dfe16d939904e98053b587f042204eef235e27cd9b5b8b6"} Oct 11 10:33:37.497376 master-1 kubenswrapper[4771]: I1011 10:33:37.497296 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:37.497376 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:37.497376 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:37.497376 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:37.497692 master-1 kubenswrapper[4771]: I1011 10:33:37.497388 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:38.342158 master-1 kubenswrapper[4771]: I1011 10:33:38.342030 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" event={"ID":"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40","Type":"ContainerStarted","Data":"db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217"} Oct 11 10:33:38.374756 master-1 kubenswrapper[4771]: I1011 10:33:38.374607 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podStartSLOduration=60.37457609 podStartE2EDuration="1m0.37457609s" podCreationTimestamp="2025-10-11 10:32:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:33:38.373168059 +0000 UTC m=+450.347394540" watchObservedRunningTime="2025-10-11 10:33:38.37457609 +0000 UTC m=+450.348802571" Oct 11 10:33:38.497486 master-1 kubenswrapper[4771]: I1011 10:33:38.497345 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:38.497486 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:38.497486 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:38.497486 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:38.497486 master-1 kubenswrapper[4771]: I1011 10:33:38.497469 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:39.497114 master-1 kubenswrapper[4771]: I1011 10:33:39.496993 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:39.497114 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:39.497114 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:39.497114 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:39.497114 master-1 kubenswrapper[4771]: I1011 10:33:39.497091 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:39.628762 master-1 kubenswrapper[4771]: I1011 10:33:39.628652 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:39.629037 master-1 kubenswrapper[4771]: I1011 10:33:39.628760 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:40.497386 master-1 kubenswrapper[4771]: I1011 10:33:40.497281 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:40.497386 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:40.497386 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:40.497386 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:40.498429 master-1 kubenswrapper[4771]: I1011 10:33:40.497394 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:41.136594 master-1 kubenswrapper[4771]: I1011 10:33:41.136471 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:41.136594 master-1 kubenswrapper[4771]: I1011 10:33:41.136587 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:41.148672 master-1 kubenswrapper[4771]: I1011 10:33:41.148580 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:41.372065 master-1 kubenswrapper[4771]: I1011 10:33:41.371956 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:33:41.498925 master-1 kubenswrapper[4771]: I1011 10:33:41.498757 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:41.498925 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:41.498925 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:41.498925 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:41.498925 master-1 kubenswrapper[4771]: I1011 10:33:41.498853 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:42.497688 master-1 kubenswrapper[4771]: I1011 10:33:42.497608 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:42.497688 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:42.497688 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:42.497688 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:42.498332 master-1 kubenswrapper[4771]: I1011 10:33:42.497706 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:43.497128 master-1 kubenswrapper[4771]: I1011 10:33:43.497034 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:43.497128 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:43.497128 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:43.497128 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:43.499906 master-1 kubenswrapper[4771]: I1011 10:33:43.497133 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:44.334872 master-1 kubenswrapper[4771]: I1011 10:33:44.334767 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/2.log" Oct 11 10:33:44.335855 master-1 kubenswrapper[4771]: I1011 10:33:44.335785 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/1.log" Oct 11 10:33:44.336476 master-1 kubenswrapper[4771]: I1011 10:33:44.336427 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-rev/0.log" Oct 11 10:33:44.337757 master-1 kubenswrapper[4771]: I1011 10:33:44.337700 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-metrics/0.log" Oct 11 10:33:44.338409 master-1 kubenswrapper[4771]: I1011 10:33:44.338330 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcdctl/0.log" Oct 11 10:33:44.340292 master-1 kubenswrapper[4771]: I1011 10:33:44.340231 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:33:44.347002 master-1 kubenswrapper[4771]: I1011 10:33:44.346928 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-master-1" oldPodUID="5268b2f2ae2aef0c7f2e7a6e651ed702" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" Oct 11 10:33:44.383835 master-1 kubenswrapper[4771]: I1011 10:33:44.383759 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/2.log" Oct 11 10:33:44.384661 master-1 kubenswrapper[4771]: I1011 10:33:44.384609 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd/1.log" Oct 11 10:33:44.385454 master-1 kubenswrapper[4771]: I1011 10:33:44.385351 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-rev/0.log" Oct 11 10:33:44.386849 master-1 kubenswrapper[4771]: I1011 10:33:44.386804 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcd-metrics/0.log" Oct 11 10:33:44.387478 master-1 kubenswrapper[4771]: I1011 10:33:44.387431 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_5268b2f2ae2aef0c7f2e7a6e651ed702/etcdctl/0.log" Oct 11 10:33:44.389092 master-1 kubenswrapper[4771]: I1011 10:33:44.389030 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e" exitCode=137 Oct 11 10:33:44.389092 master-1 kubenswrapper[4771]: I1011 10:33:44.389082 4771 generic.go:334] "Generic (PLEG): container finished" podID="5268b2f2ae2aef0c7f2e7a6e651ed702" containerID="2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8" exitCode=137 Oct 11 10:33:44.389231 master-1 kubenswrapper[4771]: I1011 10:33:44.389153 4771 scope.go:117] "RemoveContainer" containerID="8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e" Oct 11 10:33:44.389231 master-1 kubenswrapper[4771]: I1011 10:33:44.389187 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:33:44.396183 master-1 kubenswrapper[4771]: I1011 10:33:44.396118 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-master-1" oldPodUID="5268b2f2ae2aef0c7f2e7a6e651ed702" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" Oct 11 10:33:44.403145 master-1 kubenswrapper[4771]: I1011 10:33:44.402992 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-usr-local-bin\") pod \"5268b2f2ae2aef0c7f2e7a6e651ed702\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " Oct 11 10:33:44.403220 master-1 kubenswrapper[4771]: I1011 10:33:44.403178 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-resource-dir\") pod \"5268b2f2ae2aef0c7f2e7a6e651ed702\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " Oct 11 10:33:44.403420 master-1 kubenswrapper[4771]: I1011 10:33:44.403348 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-cert-dir\") pod \"5268b2f2ae2aef0c7f2e7a6e651ed702\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " Oct 11 10:33:44.403493 master-1 kubenswrapper[4771]: I1011 10:33:44.403426 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-data-dir\") pod \"5268b2f2ae2aef0c7f2e7a6e651ed702\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " Oct 11 10:33:44.403545 master-1 kubenswrapper[4771]: I1011 10:33:44.403471 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-resource-dir" (OuterVolumeSpecName: "resource-dir") pod "5268b2f2ae2aef0c7f2e7a6e651ed702" (UID: "5268b2f2ae2aef0c7f2e7a6e651ed702"). InnerVolumeSpecName "resource-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:33:44.403545 master-1 kubenswrapper[4771]: I1011 10:33:44.403454 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-usr-local-bin" (OuterVolumeSpecName: "usr-local-bin") pod "5268b2f2ae2aef0c7f2e7a6e651ed702" (UID: "5268b2f2ae2aef0c7f2e7a6e651ed702"). InnerVolumeSpecName "usr-local-bin". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:33:44.403545 master-1 kubenswrapper[4771]: I1011 10:33:44.403507 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-log-dir\") pod \"5268b2f2ae2aef0c7f2e7a6e651ed702\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " Oct 11 10:33:44.403641 master-1 kubenswrapper[4771]: I1011 10:33:44.403507 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-cert-dir" (OuterVolumeSpecName: "cert-dir") pod "5268b2f2ae2aef0c7f2e7a6e651ed702" (UID: "5268b2f2ae2aef0c7f2e7a6e651ed702"). InnerVolumeSpecName "cert-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:33:44.403641 master-1 kubenswrapper[4771]: I1011 10:33:44.403546 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-data-dir" (OuterVolumeSpecName: "data-dir") pod "5268b2f2ae2aef0c7f2e7a6e651ed702" (UID: "5268b2f2ae2aef0c7f2e7a6e651ed702"). InnerVolumeSpecName "data-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:33:44.403641 master-1 kubenswrapper[4771]: I1011 10:33:44.403552 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-log-dir" (OuterVolumeSpecName: "log-dir") pod "5268b2f2ae2aef0c7f2e7a6e651ed702" (UID: "5268b2f2ae2aef0c7f2e7a6e651ed702"). InnerVolumeSpecName "log-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:33:44.403749 master-1 kubenswrapper[4771]: I1011 10:33:44.403641 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-static-pod-dir\") pod \"5268b2f2ae2aef0c7f2e7a6e651ed702\" (UID: \"5268b2f2ae2aef0c7f2e7a6e651ed702\") " Oct 11 10:33:44.403749 master-1 kubenswrapper[4771]: I1011 10:33:44.403681 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-static-pod-dir" (OuterVolumeSpecName: "static-pod-dir") pod "5268b2f2ae2aef0c7f2e7a6e651ed702" (UID: "5268b2f2ae2aef0c7f2e7a6e651ed702"). InnerVolumeSpecName "static-pod-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:33:44.404175 master-1 kubenswrapper[4771]: I1011 10:33:44.404123 4771 reconciler_common.go:293] "Volume detached for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-static-pod-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:44.404227 master-1 kubenswrapper[4771]: I1011 10:33:44.404176 4771 reconciler_common.go:293] "Volume detached for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-usr-local-bin\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:44.404227 master-1 kubenswrapper[4771]: I1011 10:33:44.404198 4771 reconciler_common.go:293] "Volume detached for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-resource-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:44.404227 master-1 kubenswrapper[4771]: I1011 10:33:44.404217 4771 reconciler_common.go:293] "Volume detached for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-cert-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:44.404314 master-1 kubenswrapper[4771]: I1011 10:33:44.404235 4771 reconciler_common.go:293] "Volume detached for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-data-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:44.404314 master-1 kubenswrapper[4771]: I1011 10:33:44.404253 4771 reconciler_common.go:293] "Volume detached for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/5268b2f2ae2aef0c7f2e7a6e651ed702-log-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:33:44.410592 master-1 kubenswrapper[4771]: I1011 10:33:44.410539 4771 scope.go:117] "RemoveContainer" containerID="af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde" Oct 11 10:33:44.441746 master-1 kubenswrapper[4771]: I1011 10:33:44.441682 4771 scope.go:117] "RemoveContainer" containerID="958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0" Oct 11 10:33:44.447668 master-1 kubenswrapper[4771]: I1011 10:33:44.447595 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5268b2f2ae2aef0c7f2e7a6e651ed702" path="/var/lib/kubelet/pods/5268b2f2ae2aef0c7f2e7a6e651ed702/volumes" Oct 11 10:33:44.461904 master-1 kubenswrapper[4771]: I1011 10:33:44.461833 4771 scope.go:117] "RemoveContainer" containerID="cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711" Oct 11 10:33:44.486422 master-1 kubenswrapper[4771]: I1011 10:33:44.486344 4771 scope.go:117] "RemoveContainer" containerID="4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0" Oct 11 10:33:44.496907 master-1 kubenswrapper[4771]: I1011 10:33:44.496849 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:44.496907 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:44.496907 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:44.496907 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:44.497101 master-1 kubenswrapper[4771]: I1011 10:33:44.496936 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:44.507408 master-1 kubenswrapper[4771]: I1011 10:33:44.507333 4771 scope.go:117] "RemoveContainer" containerID="2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8" Oct 11 10:33:44.527221 master-1 kubenswrapper[4771]: I1011 10:33:44.527169 4771 scope.go:117] "RemoveContainer" containerID="ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05" Oct 11 10:33:44.555438 master-1 kubenswrapper[4771]: I1011 10:33:44.555378 4771 scope.go:117] "RemoveContainer" containerID="73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55" Oct 11 10:33:44.581872 master-1 kubenswrapper[4771]: I1011 10:33:44.581808 4771 scope.go:117] "RemoveContainer" containerID="7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6" Oct 11 10:33:44.610161 master-1 kubenswrapper[4771]: I1011 10:33:44.610093 4771 scope.go:117] "RemoveContainer" containerID="8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e" Oct 11 10:33:44.610796 master-1 kubenswrapper[4771]: E1011 10:33:44.610722 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e\": container with ID starting with 8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e not found: ID does not exist" containerID="8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e" Oct 11 10:33:44.610930 master-1 kubenswrapper[4771]: I1011 10:33:44.610808 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e"} err="failed to get container status \"8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e\": rpc error: code = NotFound desc = could not find container \"8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e\": container with ID starting with 8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e not found: ID does not exist" Oct 11 10:33:44.610930 master-1 kubenswrapper[4771]: I1011 10:33:44.610858 4771 scope.go:117] "RemoveContainer" containerID="af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde" Oct 11 10:33:44.611544 master-1 kubenswrapper[4771]: E1011 10:33:44.611486 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde\": container with ID starting with af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde not found: ID does not exist" containerID="af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde" Oct 11 10:33:44.611683 master-1 kubenswrapper[4771]: I1011 10:33:44.611549 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde"} err="failed to get container status \"af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde\": rpc error: code = NotFound desc = could not find container \"af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde\": container with ID starting with af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde not found: ID does not exist" Oct 11 10:33:44.611683 master-1 kubenswrapper[4771]: I1011 10:33:44.611591 4771 scope.go:117] "RemoveContainer" containerID="958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0" Oct 11 10:33:44.612073 master-1 kubenswrapper[4771]: E1011 10:33:44.611994 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0\": container with ID starting with 958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0 not found: ID does not exist" containerID="958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0" Oct 11 10:33:44.612410 master-1 kubenswrapper[4771]: I1011 10:33:44.612061 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0"} err="failed to get container status \"958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0\": rpc error: code = NotFound desc = could not find container \"958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0\": container with ID starting with 958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0 not found: ID does not exist" Oct 11 10:33:44.612410 master-1 kubenswrapper[4771]: I1011 10:33:44.612098 4771 scope.go:117] "RemoveContainer" containerID="cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711" Oct 11 10:33:44.612772 master-1 kubenswrapper[4771]: E1011 10:33:44.612703 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711\": container with ID starting with cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711 not found: ID does not exist" containerID="cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711" Oct 11 10:33:44.612772 master-1 kubenswrapper[4771]: I1011 10:33:44.612753 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711"} err="failed to get container status \"cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711\": rpc error: code = NotFound desc = could not find container \"cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711\": container with ID starting with cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711 not found: ID does not exist" Oct 11 10:33:44.612982 master-1 kubenswrapper[4771]: I1011 10:33:44.612783 4771 scope.go:117] "RemoveContainer" containerID="4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0" Oct 11 10:33:44.613310 master-1 kubenswrapper[4771]: E1011 10:33:44.613248 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0\": container with ID starting with 4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0 not found: ID does not exist" containerID="4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0" Oct 11 10:33:44.613429 master-1 kubenswrapper[4771]: I1011 10:33:44.613300 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0"} err="failed to get container status \"4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0\": rpc error: code = NotFound desc = could not find container \"4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0\": container with ID starting with 4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0 not found: ID does not exist" Oct 11 10:33:44.613429 master-1 kubenswrapper[4771]: I1011 10:33:44.613333 4771 scope.go:117] "RemoveContainer" containerID="2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8" Oct 11 10:33:44.613902 master-1 kubenswrapper[4771]: E1011 10:33:44.613799 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8\": container with ID starting with 2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8 not found: ID does not exist" containerID="2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8" Oct 11 10:33:44.613998 master-1 kubenswrapper[4771]: I1011 10:33:44.613928 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8"} err="failed to get container status \"2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8\": rpc error: code = NotFound desc = could not find container \"2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8\": container with ID starting with 2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8 not found: ID does not exist" Oct 11 10:33:44.614074 master-1 kubenswrapper[4771]: I1011 10:33:44.614008 4771 scope.go:117] "RemoveContainer" containerID="ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05" Oct 11 10:33:44.614628 master-1 kubenswrapper[4771]: E1011 10:33:44.614564 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05\": container with ID starting with ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05 not found: ID does not exist" containerID="ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05" Oct 11 10:33:44.614734 master-1 kubenswrapper[4771]: I1011 10:33:44.614627 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05"} err="failed to get container status \"ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05\": rpc error: code = NotFound desc = could not find container \"ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05\": container with ID starting with ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05 not found: ID does not exist" Oct 11 10:33:44.614734 master-1 kubenswrapper[4771]: I1011 10:33:44.614666 4771 scope.go:117] "RemoveContainer" containerID="73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55" Oct 11 10:33:44.615171 master-1 kubenswrapper[4771]: E1011 10:33:44.615117 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55\": container with ID starting with 73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55 not found: ID does not exist" containerID="73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55" Oct 11 10:33:44.615257 master-1 kubenswrapper[4771]: I1011 10:33:44.615164 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55"} err="failed to get container status \"73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55\": rpc error: code = NotFound desc = could not find container \"73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55\": container with ID starting with 73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55 not found: ID does not exist" Oct 11 10:33:44.615257 master-1 kubenswrapper[4771]: I1011 10:33:44.615192 4771 scope.go:117] "RemoveContainer" containerID="7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6" Oct 11 10:33:44.615741 master-1 kubenswrapper[4771]: E1011 10:33:44.615670 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6\": container with ID starting with 7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6 not found: ID does not exist" containerID="7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6" Oct 11 10:33:44.615741 master-1 kubenswrapper[4771]: I1011 10:33:44.615721 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6"} err="failed to get container status \"7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6\": rpc error: code = NotFound desc = could not find container \"7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6\": container with ID starting with 7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6 not found: ID does not exist" Oct 11 10:33:44.615957 master-1 kubenswrapper[4771]: I1011 10:33:44.615753 4771 scope.go:117] "RemoveContainer" containerID="8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e" Oct 11 10:33:44.616194 master-1 kubenswrapper[4771]: I1011 10:33:44.616137 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e"} err="failed to get container status \"8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e\": rpc error: code = NotFound desc = could not find container \"8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e\": container with ID starting with 8a1f303abdd5f0c02fe6a793df5fa8cce44a25f7b097770a28676ae74b39da7e not found: ID does not exist" Oct 11 10:33:44.616194 master-1 kubenswrapper[4771]: I1011 10:33:44.616166 4771 scope.go:117] "RemoveContainer" containerID="af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde" Oct 11 10:33:44.616715 master-1 kubenswrapper[4771]: I1011 10:33:44.616632 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde"} err="failed to get container status \"af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde\": rpc error: code = NotFound desc = could not find container \"af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde\": container with ID starting with af5e1ce8eeaa4f31e923faf3733d7f57ee1a57b8083addf279a4ff665cbc3fde not found: ID does not exist" Oct 11 10:33:44.616715 master-1 kubenswrapper[4771]: I1011 10:33:44.616688 4771 scope.go:117] "RemoveContainer" containerID="958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0" Oct 11 10:33:44.617223 master-1 kubenswrapper[4771]: I1011 10:33:44.617160 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0"} err="failed to get container status \"958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0\": rpc error: code = NotFound desc = could not find container \"958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0\": container with ID starting with 958716eeae4e87693f249c2e50f77614f7904dfcce99812a8c2f6b2e06fbacf0 not found: ID does not exist" Oct 11 10:33:44.617223 master-1 kubenswrapper[4771]: I1011 10:33:44.617204 4771 scope.go:117] "RemoveContainer" containerID="cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711" Oct 11 10:33:44.617724 master-1 kubenswrapper[4771]: I1011 10:33:44.617656 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711"} err="failed to get container status \"cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711\": rpc error: code = NotFound desc = could not find container \"cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711\": container with ID starting with cf1bd37b9035aa3a513b51ba1e14c267a24bb1ac86b542f0095d01337d817711 not found: ID does not exist" Oct 11 10:33:44.617724 master-1 kubenswrapper[4771]: I1011 10:33:44.617702 4771 scope.go:117] "RemoveContainer" containerID="4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0" Oct 11 10:33:44.618166 master-1 kubenswrapper[4771]: I1011 10:33:44.618107 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0"} err="failed to get container status \"4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0\": rpc error: code = NotFound desc = could not find container \"4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0\": container with ID starting with 4e485d5d5712b6cda3c4f5674c0f91abe7502ccb2d49bc78c03ccbef061a43f0 not found: ID does not exist" Oct 11 10:33:44.618166 master-1 kubenswrapper[4771]: I1011 10:33:44.618144 4771 scope.go:117] "RemoveContainer" containerID="2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8" Oct 11 10:33:44.618779 master-1 kubenswrapper[4771]: I1011 10:33:44.618698 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8"} err="failed to get container status \"2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8\": rpc error: code = NotFound desc = could not find container \"2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8\": container with ID starting with 2c883b5cc483b4b1102cd2f4e0032f04b4e86dfac92c219c11959045c43545c8 not found: ID does not exist" Oct 11 10:33:44.618877 master-1 kubenswrapper[4771]: I1011 10:33:44.618784 4771 scope.go:117] "RemoveContainer" containerID="ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05" Oct 11 10:33:44.619270 master-1 kubenswrapper[4771]: I1011 10:33:44.619206 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05"} err="failed to get container status \"ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05\": rpc error: code = NotFound desc = could not find container \"ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05\": container with ID starting with ae91d16391eadcea61802eeaddf65a6f0304c66c1e1f73804b3842b3041b8d05 not found: ID does not exist" Oct 11 10:33:44.619270 master-1 kubenswrapper[4771]: I1011 10:33:44.619254 4771 scope.go:117] "RemoveContainer" containerID="73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55" Oct 11 10:33:44.619841 master-1 kubenswrapper[4771]: I1011 10:33:44.619771 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55"} err="failed to get container status \"73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55\": rpc error: code = NotFound desc = could not find container \"73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55\": container with ID starting with 73a809cbd925d698effb0736fbe0b6f4efb7a431de066e52bb22b55ae4cc3e55 not found: ID does not exist" Oct 11 10:33:44.619841 master-1 kubenswrapper[4771]: I1011 10:33:44.619810 4771 scope.go:117] "RemoveContainer" containerID="7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6" Oct 11 10:33:44.620225 master-1 kubenswrapper[4771]: I1011 10:33:44.620163 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6"} err="failed to get container status \"7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6\": rpc error: code = NotFound desc = could not find container \"7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6\": container with ID starting with 7cb1d532ea2d89b9be52c186df240a63c25f0ae1db2bbf1085a6c556a31b0cb6 not found: ID does not exist" Oct 11 10:33:44.628574 master-1 kubenswrapper[4771]: I1011 10:33:44.628513 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:44.628716 master-1 kubenswrapper[4771]: I1011 10:33:44.628579 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:44.704506 master-1 kubenswrapper[4771]: I1011 10:33:44.704039 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-master-1" oldPodUID="5268b2f2ae2aef0c7f2e7a6e651ed702" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" Oct 11 10:33:45.497733 master-1 kubenswrapper[4771]: I1011 10:33:45.497675 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:45.497733 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:45.497733 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:45.497733 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:45.498320 master-1 kubenswrapper[4771]: I1011 10:33:45.498260 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:46.497738 master-1 kubenswrapper[4771]: I1011 10:33:46.497634 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:46.497738 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:46.497738 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:46.497738 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:46.497738 master-1 kubenswrapper[4771]: I1011 10:33:46.497728 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:47.497091 master-1 kubenswrapper[4771]: I1011 10:33:47.497010 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:47.497091 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:47.497091 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:47.497091 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:47.497552 master-1 kubenswrapper[4771]: I1011 10:33:47.497094 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:47.790029 master-1 kubenswrapper[4771]: I1011 10:33:47.789855 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:33:48.497535 master-1 kubenswrapper[4771]: I1011 10:33:48.497417 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:48.497535 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:48.497535 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:48.497535 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:48.497535 master-1 kubenswrapper[4771]: I1011 10:33:48.497509 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:49.497409 master-1 kubenswrapper[4771]: I1011 10:33:49.497286 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:49.497409 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:49.497409 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:49.497409 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:49.498351 master-1 kubenswrapper[4771]: I1011 10:33:49.497439 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:49.628571 master-1 kubenswrapper[4771]: I1011 10:33:49.628444 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:49.628936 master-1 kubenswrapper[4771]: I1011 10:33:49.628581 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:50.437070 master-1 kubenswrapper[4771]: I1011 10:33:50.436933 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:33:50.460047 master-1 kubenswrapper[4771]: I1011 10:33:50.459968 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-etcd/etcd-master-1" podUID="4bd63676-ce88-4eeb-9b7f-afaa4af19ad3" Oct 11 10:33:50.460047 master-1 kubenswrapper[4771]: I1011 10:33:50.460037 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-etcd/etcd-master-1" podUID="4bd63676-ce88-4eeb-9b7f-afaa4af19ad3" Oct 11 10:33:50.481534 master-1 kubenswrapper[4771]: I1011 10:33:50.481338 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:33:50.482678 master-1 kubenswrapper[4771]: I1011 10:33:50.482582 4771 kubelet.go:1914] "Deleted mirror pod because it is outdated" pod="openshift-etcd/etcd-master-1" Oct 11 10:33:50.491873 master-1 kubenswrapper[4771]: I1011 10:33:50.491758 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:33:50.497940 master-1 kubenswrapper[4771]: I1011 10:33:50.497883 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:50.497940 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:50.497940 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:50.497940 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:50.498784 master-1 kubenswrapper[4771]: I1011 10:33:50.497954 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:50.505896 master-1 kubenswrapper[4771]: I1011 10:33:50.505837 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:33:50.514119 master-1 kubenswrapper[4771]: I1011 10:33:50.514031 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:33:50.534449 master-1 kubenswrapper[4771]: W1011 10:33:50.534333 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1859aa05c2c75eb43d086c9ccd9c86.slice/crio-a4cfb375594ea793c8065e46bfa1f2102f58cd8b6a44fe0473d2e0310433bf19 WatchSource:0}: Error finding container a4cfb375594ea793c8065e46bfa1f2102f58cd8b6a44fe0473d2e0310433bf19: Status 404 returned error can't find the container with id a4cfb375594ea793c8065e46bfa1f2102f58cd8b6a44fe0473d2e0310433bf19 Oct 11 10:33:51.439085 master-1 kubenswrapper[4771]: I1011 10:33:51.438974 4771 generic.go:334] "Generic (PLEG): container finished" podID="2b1859aa05c2c75eb43d086c9ccd9c86" containerID="5df2d69fcce5aa4d0f872e664dab924a82b358ddfdc487a9796493b554db07ec" exitCode=0 Oct 11 10:33:51.439085 master-1 kubenswrapper[4771]: I1011 10:33:51.439050 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"2b1859aa05c2c75eb43d086c9ccd9c86","Type":"ContainerDied","Data":"5df2d69fcce5aa4d0f872e664dab924a82b358ddfdc487a9796493b554db07ec"} Oct 11 10:33:51.439564 master-1 kubenswrapper[4771]: I1011 10:33:51.439116 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"2b1859aa05c2c75eb43d086c9ccd9c86","Type":"ContainerStarted","Data":"a4cfb375594ea793c8065e46bfa1f2102f58cd8b6a44fe0473d2e0310433bf19"} Oct 11 10:33:51.496616 master-1 kubenswrapper[4771]: I1011 10:33:51.496542 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:51.496616 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:51.496616 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:51.496616 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:51.496929 master-1 kubenswrapper[4771]: I1011 10:33:51.496643 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:52.460559 master-1 kubenswrapper[4771]: I1011 10:33:52.460439 4771 generic.go:334] "Generic (PLEG): container finished" podID="2b1859aa05c2c75eb43d086c9ccd9c86" containerID="f36eed4b60a75dfc18926f5f7a62c7fe09c6ef035bfef9182c1502b7c4eeb07b" exitCode=0 Oct 11 10:33:52.460559 master-1 kubenswrapper[4771]: I1011 10:33:52.460525 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"2b1859aa05c2c75eb43d086c9ccd9c86","Type":"ContainerDied","Data":"f36eed4b60a75dfc18926f5f7a62c7fe09c6ef035bfef9182c1502b7c4eeb07b"} Oct 11 10:33:52.497794 master-1 kubenswrapper[4771]: I1011 10:33:52.497694 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:52.497794 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:52.497794 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:52.497794 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:52.498717 master-1 kubenswrapper[4771]: I1011 10:33:52.498666 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:53.471091 master-1 kubenswrapper[4771]: I1011 10:33:53.470980 4771 generic.go:334] "Generic (PLEG): container finished" podID="2b1859aa05c2c75eb43d086c9ccd9c86" containerID="0d2abececcc3750380edf401f993d45ec701aaab0b1cc115175ab53e903df0d6" exitCode=0 Oct 11 10:33:53.471091 master-1 kubenswrapper[4771]: I1011 10:33:53.471070 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"2b1859aa05c2c75eb43d086c9ccd9c86","Type":"ContainerDied","Data":"0d2abececcc3750380edf401f993d45ec701aaab0b1cc115175ab53e903df0d6"} Oct 11 10:33:53.497400 master-1 kubenswrapper[4771]: I1011 10:33:53.497278 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:53.497400 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:53.497400 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:53.497400 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:53.497802 master-1 kubenswrapper[4771]: I1011 10:33:53.497424 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:54.485900 master-1 kubenswrapper[4771]: I1011 10:33:54.485789 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"2b1859aa05c2c75eb43d086c9ccd9c86","Type":"ContainerStarted","Data":"ecbb0613c992785c9403e057fc0c874ad563e770ca35f25a2b4b2f7341f1c10c"} Oct 11 10:33:54.485900 master-1 kubenswrapper[4771]: I1011 10:33:54.485844 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"2b1859aa05c2c75eb43d086c9ccd9c86","Type":"ContainerStarted","Data":"1b08bbe8a016cc9703a454b83b5ccaac8367e55a0f3e2612f07c89255c5b066b"} Oct 11 10:33:54.485900 master-1 kubenswrapper[4771]: I1011 10:33:54.485854 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"2b1859aa05c2c75eb43d086c9ccd9c86","Type":"ContainerStarted","Data":"49bf7adabb62db980d637017833ab23f35546844d31309e50b509a3be2303a67"} Oct 11 10:33:54.497207 master-1 kubenswrapper[4771]: I1011 10:33:54.497100 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:54.497207 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:54.497207 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:54.497207 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:54.497669 master-1 kubenswrapper[4771]: I1011 10:33:54.497211 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:54.629188 master-1 kubenswrapper[4771]: I1011 10:33:54.629109 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:33:54.629333 master-1 kubenswrapper[4771]: I1011 10:33:54.629205 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:33:55.193158 master-1 kubenswrapper[4771]: I1011 10:33:55.191850 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-apiserver/apiserver-777cc846dc-qpmws"] Oct 11 10:33:55.193158 master-1 kubenswrapper[4771]: I1011 10:33:55.192768 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" containerID="cri-o://5314d6ef2281ac080baefb268e1b24e3959c52d75eecf8bba9e60d0238801c00" gracePeriod=120 Oct 11 10:33:55.194070 master-1 kubenswrapper[4771]: I1011 10:33:55.193744 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver-check-endpoints" containerID="cri-o://9b7973318d321c4747b9166204be01b90470f6b7ff6c1031063eb5d24ec05b0e" gracePeriod=120 Oct 11 10:33:55.501283 master-1 kubenswrapper[4771]: I1011 10:33:55.501094 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:55.501283 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:55.501283 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:55.501283 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:55.501283 master-1 kubenswrapper[4771]: I1011 10:33:55.501184 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:55.502639 master-1 kubenswrapper[4771]: I1011 10:33:55.502570 4771 generic.go:334] "Generic (PLEG): container finished" podID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerID="9b7973318d321c4747b9166204be01b90470f6b7ff6c1031063eb5d24ec05b0e" exitCode=0 Oct 11 10:33:55.502712 master-1 kubenswrapper[4771]: I1011 10:33:55.502661 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" event={"ID":"e2fb9636-0787-426e-bd5e-cba0ea823b2b","Type":"ContainerDied","Data":"9b7973318d321c4747b9166204be01b90470f6b7ff6c1031063eb5d24ec05b0e"} Oct 11 10:33:55.510273 master-1 kubenswrapper[4771]: I1011 10:33:55.510148 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"2b1859aa05c2c75eb43d086c9ccd9c86","Type":"ContainerStarted","Data":"2f39d1ed6551318e8799ea55ecdfbfe51ea2b9b7b26411631664f953b1d0e296"} Oct 11 10:33:55.510393 master-1 kubenswrapper[4771]: I1011 10:33:55.510324 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"2b1859aa05c2c75eb43d086c9ccd9c86","Type":"ContainerStarted","Data":"84bbf7ab3fb66f6d01d7500d037317a4cb49a3eae4199b8937858e7e953c7fd3"} Oct 11 10:33:55.552629 master-1 kubenswrapper[4771]: I1011 10:33:55.552295 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-etcd/etcd-master-1" podStartSLOduration=5.552265338 podStartE2EDuration="5.552265338s" podCreationTimestamp="2025-10-11 10:33:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:33:55.550437464 +0000 UTC m=+467.524663935" watchObservedRunningTime="2025-10-11 10:33:55.552265338 +0000 UTC m=+467.526491809" Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: I1011 10:33:56.066539 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:33:56.066643 master-1 kubenswrapper[4771]: I1011 10:33:56.066620 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:56.497212 master-1 kubenswrapper[4771]: I1011 10:33:56.497046 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:56.497212 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:56.497212 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:56.497212 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:56.497212 master-1 kubenswrapper[4771]: I1011 10:33:56.497146 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:57.498032 master-1 kubenswrapper[4771]: I1011 10:33:57.497964 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:57.498032 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:57.498032 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:57.498032 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:57.499254 master-1 kubenswrapper[4771]: I1011 10:33:57.498056 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:58.498052 master-1 kubenswrapper[4771]: I1011 10:33:58.497906 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:58.498052 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:58.498052 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:58.498052 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:58.499263 master-1 kubenswrapper[4771]: I1011 10:33:58.498057 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:59.496791 master-1 kubenswrapper[4771]: I1011 10:33:59.496671 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:33:59.496791 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:33:59.496791 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:33:59.496791 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:33:59.496791 master-1 kubenswrapper[4771]: I1011 10:33:59.496782 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:33:59.661752 master-1 kubenswrapper[4771]: I1011 10:33:59.661696 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:34:00.498698 master-1 kubenswrapper[4771]: I1011 10:34:00.498564 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:00.498698 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:00.498698 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:00.498698 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:00.498698 master-1 kubenswrapper[4771]: I1011 10:34:00.498666 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:00.506974 master-1 kubenswrapper[4771]: I1011 10:34:00.506886 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-etcd/etcd-master-1" Oct 11 10:34:00.506974 master-1 kubenswrapper[4771]: I1011 10:34:00.506977 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-etcd/etcd-master-1" Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: I1011 10:34:01.066649 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:01.066699 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:01.068253 master-1 kubenswrapper[4771]: I1011 10:34:01.067818 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:01.497292 master-1 kubenswrapper[4771]: I1011 10:34:01.497083 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:01.497292 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:01.497292 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:01.497292 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:01.497292 master-1 kubenswrapper[4771]: I1011 10:34:01.497183 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:02.498161 master-1 kubenswrapper[4771]: I1011 10:34:02.498055 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:02.498161 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:02.498161 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:02.498161 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:02.498161 master-1 kubenswrapper[4771]: I1011 10:34:02.498141 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:03.497085 master-1 kubenswrapper[4771]: I1011 10:34:03.496975 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:03.497085 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:03.497085 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:03.497085 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:03.497085 master-1 kubenswrapper[4771]: I1011 10:34:03.497052 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:04.496702 master-1 kubenswrapper[4771]: I1011 10:34:04.496624 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:04.496702 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:04.496702 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:04.496702 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:04.496702 master-1 kubenswrapper[4771]: I1011 10:34:04.496697 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:05.497686 master-1 kubenswrapper[4771]: I1011 10:34:05.497569 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:05.497686 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:05.497686 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:05.497686 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:05.497686 master-1 kubenswrapper[4771]: I1011 10:34:05.497645 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: I1011 10:34:06.067198 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:06.067284 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:06.068284 master-1 kubenswrapper[4771]: I1011 10:34:06.067308 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:06.068284 master-1 kubenswrapper[4771]: I1011 10:34:06.067569 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:34:06.497961 master-1 kubenswrapper[4771]: I1011 10:34:06.497739 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:06.497961 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:06.497961 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:06.497961 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:06.497961 master-1 kubenswrapper[4771]: I1011 10:34:06.497859 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:07.497569 master-1 kubenswrapper[4771]: I1011 10:34:07.497460 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:07.497569 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:07.497569 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:07.497569 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:07.497569 master-1 kubenswrapper[4771]: I1011 10:34:07.497569 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:08.498140 master-1 kubenswrapper[4771]: I1011 10:34:08.498011 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:08.498140 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:08.498140 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:08.498140 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:08.499617 master-1 kubenswrapper[4771]: I1011 10:34:08.498143 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:09.497870 master-1 kubenswrapper[4771]: I1011 10:34:09.497730 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:09.497870 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:09.497870 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:09.497870 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:09.497870 master-1 kubenswrapper[4771]: I1011 10:34:09.497836 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:10.497505 master-1 kubenswrapper[4771]: I1011 10:34:10.497336 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:10.497505 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:10.497505 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:10.497505 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:10.497505 master-1 kubenswrapper[4771]: I1011 10:34:10.497472 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:10.526574 master-1 kubenswrapper[4771]: I1011 10:34:10.526447 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-etcd/etcd-master-1" Oct 11 10:34:10.543950 master-1 kubenswrapper[4771]: I1011 10:34:10.543883 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-etcd/etcd-master-1" Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: I1011 10:34:11.066294 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:11.066396 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:11.067920 master-1 kubenswrapper[4771]: I1011 10:34:11.066408 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:11.497672 master-1 kubenswrapper[4771]: I1011 10:34:11.497508 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:11.497672 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:11.497672 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:11.497672 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:11.497672 master-1 kubenswrapper[4771]: I1011 10:34:11.497598 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:12.497574 master-1 kubenswrapper[4771]: I1011 10:34:12.497480 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:12.497574 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:12.497574 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:12.497574 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:12.497574 master-1 kubenswrapper[4771]: I1011 10:34:12.497560 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:13.498106 master-1 kubenswrapper[4771]: I1011 10:34:13.497973 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:13.498106 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:13.498106 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:13.498106 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:13.499449 master-1 kubenswrapper[4771]: I1011 10:34:13.498098 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:14.497077 master-1 kubenswrapper[4771]: I1011 10:34:14.496997 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:14.497077 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:14.497077 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:14.497077 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:14.497077 master-1 kubenswrapper[4771]: I1011 10:34:14.497086 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:15.420022 master-1 kubenswrapper[4771]: I1011 10:34:15.419936 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/installer-2-master-1"] Oct 11 10:34:15.420753 master-1 kubenswrapper[4771]: I1011 10:34:15.420610 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.435564 master-1 kubenswrapper[4771]: I1011 10:34:15.435468 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-2-master-1"] Oct 11 10:34:15.497195 master-1 kubenswrapper[4771]: I1011 10:34:15.497085 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:15.497195 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:15.497195 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:15.497195 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:15.497195 master-1 kubenswrapper[4771]: I1011 10:34:15.497186 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:15.534569 master-1 kubenswrapper[4771]: I1011 10:34:15.534445 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-var-lock\") pod \"installer-2-master-1\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.534904 master-1 kubenswrapper[4771]: I1011 10:34:15.534765 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kube-api-access\") pod \"installer-2-master-1\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.534904 master-1 kubenswrapper[4771]: I1011 10:34:15.534807 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kubelet-dir\") pod \"installer-2-master-1\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.636789 master-1 kubenswrapper[4771]: I1011 10:34:15.636669 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kube-api-access\") pod \"installer-2-master-1\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.636789 master-1 kubenswrapper[4771]: I1011 10:34:15.636755 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kubelet-dir\") pod \"installer-2-master-1\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.636789 master-1 kubenswrapper[4771]: I1011 10:34:15.636809 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-var-lock\") pod \"installer-2-master-1\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.637268 master-1 kubenswrapper[4771]: I1011 10:34:15.636977 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-var-lock\") pod \"installer-2-master-1\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.637268 master-1 kubenswrapper[4771]: I1011 10:34:15.636973 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kubelet-dir\") pod \"installer-2-master-1\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.664663 master-1 kubenswrapper[4771]: I1011 10:34:15.664608 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kube-api-access\") pod \"installer-2-master-1\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:15.748584 master-1 kubenswrapper[4771]: I1011 10:34:15.748343 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: I1011 10:34:16.065698 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:16.065757 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:16.067041 master-1 kubenswrapper[4771]: I1011 10:34:16.065779 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:16.230261 master-1 kubenswrapper[4771]: I1011 10:34:16.229800 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-2-master-1"] Oct 11 10:34:16.497855 master-1 kubenswrapper[4771]: I1011 10:34:16.497743 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:34:16.497855 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:34:16.497855 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:34:16.497855 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:34:16.499089 master-1 kubenswrapper[4771]: I1011 10:34:16.497863 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:16.499089 master-1 kubenswrapper[4771]: I1011 10:34:16.497950 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:34:16.499282 master-1 kubenswrapper[4771]: I1011 10:34:16.499106 4771 kuberuntime_manager.go:1027] "Message for Container of pod" containerName="router" containerStatusID={"Type":"cri-o","ID":"2fd6e0cb14ecdcadbf2571f6d4dd1d2a4a1e6cf999fc333d09b9fc98b284b780"} pod="openshift-ingress/router-default-5ddb89f76-z5t6x" containerMessage="Container router failed startup probe, will be restarted" Oct 11 10:34:16.499282 master-1 kubenswrapper[4771]: I1011 10:34:16.499171 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" containerID="cri-o://2fd6e0cb14ecdcadbf2571f6d4dd1d2a4a1e6cf999fc333d09b9fc98b284b780" gracePeriod=3600 Oct 11 10:34:16.655268 master-1 kubenswrapper[4771]: I1011 10:34:16.655205 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-2-master-1" event={"ID":"c1c3b2b9-8880-496b-88ed-9706cd8ee23d","Type":"ContainerStarted","Data":"781880873ca29705a429a8abc16c37af29927d033898ec8fedabee8745269269"} Oct 11 10:34:17.661817 master-1 kubenswrapper[4771]: I1011 10:34:17.661735 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-2-master-1" event={"ID":"c1c3b2b9-8880-496b-88ed-9706cd8ee23d","Type":"ContainerStarted","Data":"d5b95693856e76475228e56822cf59bc988be47b5715c02d7d3f81ff2fa1bb74"} Oct 11 10:34:17.688328 master-1 kubenswrapper[4771]: I1011 10:34:17.688207 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/installer-2-master-1" podStartSLOduration=2.68818185 podStartE2EDuration="2.68818185s" podCreationTimestamp="2025-10-11 10:34:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:34:17.683469381 +0000 UTC m=+489.657695892" watchObservedRunningTime="2025-10-11 10:34:17.68818185 +0000 UTC m=+489.662408331" Oct 11 10:34:17.786987 master-1 kubenswrapper[4771]: I1011 10:34:17.786892 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: I1011 10:34:21.064775 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:21.064871 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:21.065813 master-1 kubenswrapper[4771]: I1011 10:34:21.064878 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: I1011 10:34:26.064460 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:26.064583 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:26.066589 master-1 kubenswrapper[4771]: I1011 10:34:26.064583 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: I1011 10:34:31.066643 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:31.066752 master-1 kubenswrapper[4771]: I1011 10:34:31.066733 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: I1011 10:34:36.064396 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:36.064541 master-1 kubenswrapper[4771]: I1011 10:34:36.064491 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: I1011 10:34:41.064667 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:41.064815 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:41.067169 master-1 kubenswrapper[4771]: I1011 10:34:41.067119 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:34:46.059561 master-1 kubenswrapper[4771]: I1011 10:34:46.059432 4771 patch_prober.go:28] interesting pod/apiserver-777cc846dc-qpmws container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.48:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.48:8443: connect: connection refused" start-of-body= Oct 11 10:34:46.059561 master-1 kubenswrapper[4771]: I1011 10:34:46.059552 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.48:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.48:8443: connect: connection refused" Oct 11 10:34:46.861094 master-1 kubenswrapper[4771]: I1011 10:34:46.861028 4771 generic.go:334] "Generic (PLEG): container finished" podID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerID="5314d6ef2281ac080baefb268e1b24e3959c52d75eecf8bba9e60d0238801c00" exitCode=0 Oct 11 10:34:46.861311 master-1 kubenswrapper[4771]: I1011 10:34:46.861096 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" event={"ID":"e2fb9636-0787-426e-bd5e-cba0ea823b2b","Type":"ContainerDied","Data":"5314d6ef2281ac080baefb268e1b24e3959c52d75eecf8bba9e60d0238801c00"} Oct 11 10:34:46.861311 master-1 kubenswrapper[4771]: I1011 10:34:46.861143 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" event={"ID":"e2fb9636-0787-426e-bd5e-cba0ea823b2b","Type":"ContainerDied","Data":"a41a821c8fbcdc8c024fe125a36dfc655949ba099ab1bab4420d6e97047ce118"} Oct 11 10:34:46.861311 master-1 kubenswrapper[4771]: I1011 10:34:46.861168 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a41a821c8fbcdc8c024fe125a36dfc655949ba099ab1bab4420d6e97047ce118" Oct 11 10:34:46.876501 master-1 kubenswrapper[4771]: I1011 10:34:46.876282 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:34:46.926438 master-1 kubenswrapper[4771]: I1011 10:34:46.926321 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-apiserver/apiserver-7845cf54d8-g8x5z"] Oct 11 10:34:46.926888 master-1 kubenswrapper[4771]: E1011 10:34:46.926822 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" Oct 11 10:34:46.926888 master-1 kubenswrapper[4771]: I1011 10:34:46.926877 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" Oct 11 10:34:46.927059 master-1 kubenswrapper[4771]: E1011 10:34:46.926903 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="fix-audit-permissions" Oct 11 10:34:46.927059 master-1 kubenswrapper[4771]: I1011 10:34:46.926953 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="fix-audit-permissions" Oct 11 10:34:46.927059 master-1 kubenswrapper[4771]: E1011 10:34:46.926979 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver-check-endpoints" Oct 11 10:34:46.927059 master-1 kubenswrapper[4771]: I1011 10:34:46.926998 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver-check-endpoints" Oct 11 10:34:46.927329 master-1 kubenswrapper[4771]: I1011 10:34:46.927230 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver" Oct 11 10:34:46.927329 master-1 kubenswrapper[4771]: I1011 10:34:46.927262 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" containerName="openshift-apiserver-check-endpoints" Oct 11 10:34:46.928722 master-1 kubenswrapper[4771]: I1011 10:34:46.928670 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:46.936847 master-1 kubenswrapper[4771]: I1011 10:34:46.936786 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-apiserver/apiserver-7845cf54d8-g8x5z"] Oct 11 10:34:46.961560 master-1 kubenswrapper[4771]: I1011 10:34:46.961435 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.961869 master-1 kubenswrapper[4771]: I1011 10:34:46.961638 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpdjh\" (UniqueName: \"kubernetes.io/projected/e2fb9636-0787-426e-bd5e-cba0ea823b2b-kube-api-access-dpdjh\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.961869 master-1 kubenswrapper[4771]: I1011 10:34:46.961755 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-client\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.961869 master-1 kubenswrapper[4771]: I1011 10:34:46.961835 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-image-import-ca\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.962214 master-1 kubenswrapper[4771]: I1011 10:34:46.961882 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit" (OuterVolumeSpecName: "audit") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "audit". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:34:46.962214 master-1 kubenswrapper[4771]: I1011 10:34:46.961904 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-trusted-ca-bundle\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.962214 master-1 kubenswrapper[4771]: I1011 10:34:46.961981 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-serving-ca\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.962214 master-1 kubenswrapper[4771]: I1011 10:34:46.962035 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-serving-cert\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.962214 master-1 kubenswrapper[4771]: I1011 10:34:46.962113 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-node-pullsecrets\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.962577 master-1 kubenswrapper[4771]: I1011 10:34:46.962216 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-config\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.962577 master-1 kubenswrapper[4771]: I1011 10:34:46.962378 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-image-import-ca" (OuterVolumeSpecName: "image-import-ca") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "image-import-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:34:46.962577 master-1 kubenswrapper[4771]: I1011 10:34:46.962401 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit-dir\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.962577 master-1 kubenswrapper[4771]: I1011 10:34:46.962507 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-encryption-config\") pod \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\" (UID: \"e2fb9636-0787-426e-bd5e-cba0ea823b2b\") " Oct 11 10:34:46.962700 master-1 kubenswrapper[4771]: I1011 10:34:46.962618 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-node-pullsecrets" (OuterVolumeSpecName: "node-pullsecrets") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "node-pullsecrets". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:34:46.962747 master-1 kubenswrapper[4771]: I1011 10:34:46.962730 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:34:46.963075 master-1 kubenswrapper[4771]: I1011 10:34:46.963031 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:46.963075 master-1 kubenswrapper[4771]: I1011 10:34:46.963070 4771 reconciler_common.go:293] "Volume detached for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-audit\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:46.963142 master-1 kubenswrapper[4771]: I1011 10:34:46.963123 4771 reconciler_common.go:293] "Volume detached for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-image-import-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:46.963182 master-1 kubenswrapper[4771]: I1011 10:34:46.963146 4771 reconciler_common.go:293] "Volume detached for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/e2fb9636-0787-426e-bd5e-cba0ea823b2b-node-pullsecrets\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:46.963320 master-1 kubenswrapper[4771]: I1011 10:34:46.963295 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-serving-ca" (OuterVolumeSpecName: "etcd-serving-ca") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "etcd-serving-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:34:46.963443 master-1 kubenswrapper[4771]: I1011 10:34:46.963350 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-config" (OuterVolumeSpecName: "config") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:34:46.963833 master-1 kubenswrapper[4771]: I1011 10:34:46.963754 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:34:46.965633 master-1 kubenswrapper[4771]: I1011 10:34:46.965541 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fb9636-0787-426e-bd5e-cba0ea823b2b-kube-api-access-dpdjh" (OuterVolumeSpecName: "kube-api-access-dpdjh") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "kube-api-access-dpdjh". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:34:46.966855 master-1 kubenswrapper[4771]: I1011 10:34:46.966814 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:34:46.967994 master-1 kubenswrapper[4771]: I1011 10:34:46.967940 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-encryption-config" (OuterVolumeSpecName: "encryption-config") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "encryption-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:34:46.974863 master-1 kubenswrapper[4771]: I1011 10:34:46.974807 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-client" (OuterVolumeSpecName: "etcd-client") pod "e2fb9636-0787-426e-bd5e-cba0ea823b2b" (UID: "e2fb9636-0787-426e-bd5e-cba0ea823b2b"). InnerVolumeSpecName "etcd-client". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:34:47.065158 master-1 kubenswrapper[4771]: I1011 10:34:47.065073 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit-dir\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065177 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-serving-ca\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065216 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-encryption-config\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065390 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-image-import-ca\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065515 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-client\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065583 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-node-pullsecrets\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065613 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b78cw\" (UniqueName: \"kubernetes.io/projected/a2bf529d-094c-4406-8ce6-890cf8c0b840-kube-api-access-b78cw\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065641 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065665 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-serving-cert\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065686 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-trusted-ca-bundle\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065723 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-config\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065840 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-dpdjh\" (UniqueName: \"kubernetes.io/projected/e2fb9636-0787-426e-bd5e-cba0ea823b2b-kube-api-access-dpdjh\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065858 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-client\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065872 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:47.065957 master-1 kubenswrapper[4771]: I1011 10:34:47.065885 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-etcd-serving-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:47.066486 master-1 kubenswrapper[4771]: I1011 10:34:47.066009 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:47.066486 master-1 kubenswrapper[4771]: I1011 10:34:47.066087 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2fb9636-0787-426e-bd5e-cba0ea823b2b-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:47.066486 master-1 kubenswrapper[4771]: I1011 10:34:47.066109 4771 reconciler_common.go:293] "Volume detached for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/e2fb9636-0787-426e-bd5e-cba0ea823b2b-encryption-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:47.167686 master-1 kubenswrapper[4771]: I1011 10:34:47.167500 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-image-import-ca\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.167686 master-1 kubenswrapper[4771]: I1011 10:34:47.167590 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-client\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.167686 master-1 kubenswrapper[4771]: I1011 10:34:47.167626 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-node-pullsecrets\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.167686 master-1 kubenswrapper[4771]: I1011 10:34:47.167648 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-b78cw\" (UniqueName: \"kubernetes.io/projected/a2bf529d-094c-4406-8ce6-890cf8c0b840-kube-api-access-b78cw\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.167686 master-1 kubenswrapper[4771]: I1011 10:34:47.167671 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.167686 master-1 kubenswrapper[4771]: I1011 10:34:47.167691 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-trusted-ca-bundle\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.167686 master-1 kubenswrapper[4771]: I1011 10:34:47.167707 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-serving-cert\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.168172 master-1 kubenswrapper[4771]: I1011 10:34:47.167734 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-config\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.168172 master-1 kubenswrapper[4771]: I1011 10:34:47.167758 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit-dir\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.168172 master-1 kubenswrapper[4771]: I1011 10:34:47.167794 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-serving-ca\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.168172 master-1 kubenswrapper[4771]: I1011 10:34:47.167816 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-encryption-config\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.168172 master-1 kubenswrapper[4771]: I1011 10:34:47.167868 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-node-pullsecrets\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.168172 master-1 kubenswrapper[4771]: I1011 10:34:47.168111 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit-dir\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.169157 master-1 kubenswrapper[4771]: I1011 10:34:47.168770 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-image-import-ca\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.169157 master-1 kubenswrapper[4771]: I1011 10:34:47.169014 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-config\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.169157 master-1 kubenswrapper[4771]: I1011 10:34:47.169109 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.169827 master-1 kubenswrapper[4771]: I1011 10:34:47.169776 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-serving-ca\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.170806 master-1 kubenswrapper[4771]: I1011 10:34:47.170718 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-trusted-ca-bundle\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.171577 master-1 kubenswrapper[4771]: I1011 10:34:47.171521 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-encryption-config\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.173483 master-1 kubenswrapper[4771]: I1011 10:34:47.173426 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-serving-cert\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.173827 master-1 kubenswrapper[4771]: I1011 10:34:47.173777 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-client\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.199986 master-1 kubenswrapper[4771]: I1011 10:34:47.199918 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-b78cw\" (UniqueName: \"kubernetes.io/projected/a2bf529d-094c-4406-8ce6-890cf8c0b840-kube-api-access-b78cw\") pod \"apiserver-7845cf54d8-g8x5z\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.249823 master-1 kubenswrapper[4771]: I1011 10:34:47.249740 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:47.724746 master-1 kubenswrapper[4771]: I1011 10:34:47.724612 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-apiserver/apiserver-7845cf54d8-g8x5z"] Oct 11 10:34:47.730860 master-1 kubenswrapper[4771]: W1011 10:34:47.730780 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bf529d_094c_4406_8ce6_890cf8c0b840.slice/crio-e398827cf779d365dfc4e6c2443dd2f776caa9a8ba75c41d00aafc513ef28957 WatchSource:0}: Error finding container e398827cf779d365dfc4e6c2443dd2f776caa9a8ba75c41d00aafc513ef28957: Status 404 returned error can't find the container with id e398827cf779d365dfc4e6c2443dd2f776caa9a8ba75c41d00aafc513ef28957 Oct 11 10:34:47.796698 master-1 kubenswrapper[4771]: I1011 10:34:47.796621 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:34:47.872946 master-1 kubenswrapper[4771]: I1011 10:34:47.872856 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" event={"ID":"a2bf529d-094c-4406-8ce6-890cf8c0b840","Type":"ContainerStarted","Data":"e398827cf779d365dfc4e6c2443dd2f776caa9a8ba75c41d00aafc513ef28957"} Oct 11 10:34:47.872946 master-1 kubenswrapper[4771]: I1011 10:34:47.872917 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-777cc846dc-qpmws" Oct 11 10:34:47.927150 master-1 kubenswrapper[4771]: I1011 10:34:47.927076 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-apiserver/apiserver-777cc846dc-qpmws"] Oct 11 10:34:47.930165 master-1 kubenswrapper[4771]: I1011 10:34:47.930072 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-apiserver/apiserver-777cc846dc-qpmws"] Oct 11 10:34:48.447462 master-1 kubenswrapper[4771]: I1011 10:34:48.447328 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2fb9636-0787-426e-bd5e-cba0ea823b2b" path="/var/lib/kubelet/pods/e2fb9636-0787-426e-bd5e-cba0ea823b2b/volumes" Oct 11 10:34:48.881563 master-1 kubenswrapper[4771]: I1011 10:34:48.881469 4771 generic.go:334] "Generic (PLEG): container finished" podID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerID="5a44ec551f4491e724d147c13cc98b993a3968bac1f8f715ba1d91a8129c8004" exitCode=0 Oct 11 10:34:48.881563 master-1 kubenswrapper[4771]: I1011 10:34:48.881554 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" event={"ID":"a2bf529d-094c-4406-8ce6-890cf8c0b840","Type":"ContainerDied","Data":"5a44ec551f4491e724d147c13cc98b993a3968bac1f8f715ba1d91a8129c8004"} Oct 11 10:34:49.891467 master-1 kubenswrapper[4771]: I1011 10:34:49.891392 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" event={"ID":"a2bf529d-094c-4406-8ce6-890cf8c0b840","Type":"ContainerStarted","Data":"a0772db7a40ce6f228f65f235a6668a5f2f1781a4f227000cf9ad01206d856f2"} Oct 11 10:34:49.892318 master-1 kubenswrapper[4771]: I1011 10:34:49.891478 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" event={"ID":"a2bf529d-094c-4406-8ce6-890cf8c0b840","Type":"ContainerStarted","Data":"2ccd5ea4ca8c2b32e04ef7419d2c1c1ac0971dd1b18e1a37cd16058b70e5a98c"} Oct 11 10:34:49.923786 master-1 kubenswrapper[4771]: I1011 10:34:49.923716 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podStartSLOduration=54.923697914 podStartE2EDuration="54.923697914s" podCreationTimestamp="2025-10-11 10:33:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:34:49.92219166 +0000 UTC m=+521.896418141" watchObservedRunningTime="2025-10-11 10:34:49.923697914 +0000 UTC m=+521.897924365" Oct 11 10:34:52.250682 master-1 kubenswrapper[4771]: I1011 10:34:52.250592 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:52.250682 master-1 kubenswrapper[4771]: I1011 10:34:52.250700 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:52.262800 master-1 kubenswrapper[4771]: I1011 10:34:52.262740 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:52.922096 master-1 kubenswrapper[4771]: I1011 10:34:52.922032 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:34:54.821447 master-1 kubenswrapper[4771]: I1011 10:34:54.821344 4771 kubelet.go:2431] "SyncLoop REMOVE" source="file" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:34:54.822291 master-1 kubenswrapper[4771]: I1011 10:34:54.821963 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver" containerID="cri-o://7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea" gracePeriod=135 Oct 11 10:34:54.822291 master-1 kubenswrapper[4771]: I1011 10:34:54.822017 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-check-endpoints" containerID="cri-o://49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd" gracePeriod=135 Oct 11 10:34:54.822291 master-1 kubenswrapper[4771]: I1011 10:34:54.822041 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-cert-regeneration-controller" containerID="cri-o://d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10" gracePeriod=135 Oct 11 10:34:54.822291 master-1 kubenswrapper[4771]: I1011 10:34:54.822131 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-insecure-readyz" containerID="cri-o://3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7" gracePeriod=135 Oct 11 10:34:54.822716 master-1 kubenswrapper[4771]: I1011 10:34:54.822328 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-cert-syncer" containerID="cri-o://c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1" gracePeriod=135 Oct 11 10:34:54.825037 master-1 kubenswrapper[4771]: I1011 10:34:54.824970 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:34:54.825324 master-1 kubenswrapper[4771]: E1011 10:34:54.825272 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="setup" Oct 11 10:34:54.825324 master-1 kubenswrapper[4771]: I1011 10:34:54.825304 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="setup" Oct 11 10:34:54.825324 master-1 kubenswrapper[4771]: E1011 10:34:54.825320 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver" Oct 11 10:34:54.825324 master-1 kubenswrapper[4771]: I1011 10:34:54.825333 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: E1011 10:34:54.825351 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-cert-regeneration-controller" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: I1011 10:34:54.825371 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-cert-regeneration-controller" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: E1011 10:34:54.825438 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-cert-syncer" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: I1011 10:34:54.825451 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-cert-syncer" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: E1011 10:34:54.825468 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-check-endpoints" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: I1011 10:34:54.825482 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-check-endpoints" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: E1011 10:34:54.825498 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-insecure-readyz" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: I1011 10:34:54.825511 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-insecure-readyz" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: I1011 10:34:54.825642 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-cert-regeneration-controller" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: I1011 10:34:54.825663 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: I1011 10:34:54.825682 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-check-endpoints" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: I1011 10:34:54.825704 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-cert-syncer" Oct 11 10:34:54.825736 master-1 kubenswrapper[4771]: I1011 10:34:54.825716 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="34b1362996d1e0c2cea0bee73eb18468" containerName="kube-apiserver-insecure-readyz" Oct 11 10:34:54.873118 master-1 kubenswrapper[4771]: I1011 10:34:54.873031 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:34:54.873466 master-1 kubenswrapper[4771]: I1011 10:34:54.873201 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:34:54.873466 master-1 kubenswrapper[4771]: I1011 10:34:54.873335 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:34:54.990778 master-1 kubenswrapper[4771]: I1011 10:34:54.990641 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:34:54.990778 master-1 kubenswrapper[4771]: I1011 10:34:54.990758 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:34:54.990962 master-1 kubenswrapper[4771]: I1011 10:34:54.990833 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:34:54.990962 master-1 kubenswrapper[4771]: I1011 10:34:54.990837 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:34:54.991075 master-1 kubenswrapper[4771]: I1011 10:34:54.991015 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:34:54.991166 master-1 kubenswrapper[4771]: I1011 10:34:54.991100 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:34:55.933727 master-1 kubenswrapper[4771]: I1011 10:34:55.933635 4771 generic.go:334] "Generic (PLEG): container finished" podID="c1c3b2b9-8880-496b-88ed-9706cd8ee23d" containerID="d5b95693856e76475228e56822cf59bc988be47b5715c02d7d3f81ff2fa1bb74" exitCode=0 Oct 11 10:34:55.934389 master-1 kubenswrapper[4771]: I1011 10:34:55.933725 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-2-master-1" event={"ID":"c1c3b2b9-8880-496b-88ed-9706cd8ee23d","Type":"ContainerDied","Data":"d5b95693856e76475228e56822cf59bc988be47b5715c02d7d3f81ff2fa1bb74"} Oct 11 10:34:55.940641 master-1 kubenswrapper[4771]: I1011 10:34:55.940574 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_34b1362996d1e0c2cea0bee73eb18468/kube-apiserver-cert-syncer/0.log" Oct 11 10:34:55.941981 master-1 kubenswrapper[4771]: I1011 10:34:55.941940 4771 generic.go:334] "Generic (PLEG): container finished" podID="34b1362996d1e0c2cea0bee73eb18468" containerID="49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd" exitCode=0 Oct 11 10:34:55.941981 master-1 kubenswrapper[4771]: I1011 10:34:55.941977 4771 generic.go:334] "Generic (PLEG): container finished" podID="34b1362996d1e0c2cea0bee73eb18468" containerID="3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7" exitCode=0 Oct 11 10:34:55.942197 master-1 kubenswrapper[4771]: I1011 10:34:55.941994 4771 generic.go:334] "Generic (PLEG): container finished" podID="34b1362996d1e0c2cea0bee73eb18468" containerID="d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10" exitCode=0 Oct 11 10:34:55.942197 master-1 kubenswrapper[4771]: I1011 10:34:55.942012 4771 generic.go:334] "Generic (PLEG): container finished" podID="34b1362996d1e0c2cea0bee73eb18468" containerID="c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1" exitCode=2 Oct 11 10:34:55.963847 master-1 kubenswrapper[4771]: I1011 10:34:55.963643 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="34b1362996d1e0c2cea0bee73eb18468" podUID="e39186c2ebd02622803bdbec6984de2a" Oct 11 10:34:56.028805 master-1 kubenswrapper[4771]: E1011 10:34:56.028693 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" podUID="d7647696-42d9-4dd9-bc3b-a4d52a42cf9a" Oct 11 10:34:56.028805 master-1 kubenswrapper[4771]: E1011 10:34:56.028718 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" podUID="6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b" Oct 11 10:34:56.948525 master-1 kubenswrapper[4771]: I1011 10:34:56.948435 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:34:56.949342 master-1 kubenswrapper[4771]: I1011 10:34:56.948547 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:34:57.329398 master-1 kubenswrapper[4771]: I1011 10:34:57.329313 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:57.425011 master-1 kubenswrapper[4771]: I1011 10:34:57.424899 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kubelet-dir\") pod \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " Oct 11 10:34:57.425357 master-1 kubenswrapper[4771]: I1011 10:34:57.425125 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-var-lock\") pod \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " Oct 11 10:34:57.425357 master-1 kubenswrapper[4771]: I1011 10:34:57.425116 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "c1c3b2b9-8880-496b-88ed-9706cd8ee23d" (UID: "c1c3b2b9-8880-496b-88ed-9706cd8ee23d"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:34:57.425357 master-1 kubenswrapper[4771]: I1011 10:34:57.425232 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kube-api-access\") pod \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\" (UID: \"c1c3b2b9-8880-496b-88ed-9706cd8ee23d\") " Oct 11 10:34:57.425357 master-1 kubenswrapper[4771]: I1011 10:34:57.425265 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-var-lock" (OuterVolumeSpecName: "var-lock") pod "c1c3b2b9-8880-496b-88ed-9706cd8ee23d" (UID: "c1c3b2b9-8880-496b-88ed-9706cd8ee23d"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:34:57.425656 master-1 kubenswrapper[4771]: I1011 10:34:57.425582 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:34:57.425823 master-1 kubenswrapper[4771]: E1011 10:34:57.425781 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:36:59.42575903 +0000 UTC m=+651.399985481 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:34:57.425919 master-1 kubenswrapper[4771]: I1011 10:34:57.425829 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:57.425919 master-1 kubenswrapper[4771]: I1011 10:34:57.425850 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:57.429513 master-1 kubenswrapper[4771]: I1011 10:34:57.429415 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "c1c3b2b9-8880-496b-88ed-9706cd8ee23d" (UID: "c1c3b2b9-8880-496b-88ed-9706cd8ee23d"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:34:57.529275 master-1 kubenswrapper[4771]: I1011 10:34:57.529084 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:34:57.529581 master-1 kubenswrapper[4771]: I1011 10:34:57.529315 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c1c3b2b9-8880-496b-88ed-9706cd8ee23d-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:34:57.529581 master-1 kubenswrapper[4771]: E1011 10:34:57.529339 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:36:59.52929247 +0000 UTC m=+651.503518941 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:34:57.731883 master-1 kubenswrapper[4771]: I1011 10:34:57.731820 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/metrics-server-65d86dff78-bg7lk"] Oct 11 10:34:57.733008 master-1 kubenswrapper[4771]: I1011 10:34:57.732948 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" containerID="cri-o://d71774e5747fba198d1f1c685867c43372766be8110c50262b34cb5aee247b7d" gracePeriod=170 Oct 11 10:34:57.955757 master-1 kubenswrapper[4771]: I1011 10:34:57.955676 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-2-master-1" event={"ID":"c1c3b2b9-8880-496b-88ed-9706cd8ee23d","Type":"ContainerDied","Data":"781880873ca29705a429a8abc16c37af29927d033898ec8fedabee8745269269"} Oct 11 10:34:57.955757 master-1 kubenswrapper[4771]: I1011 10:34:57.955750 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="781880873ca29705a429a8abc16c37af29927d033898ec8fedabee8745269269" Oct 11 10:34:57.956676 master-1 kubenswrapper[4771]: I1011 10:34:57.956523 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-2-master-1" Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: I1011 10:34:58.245601 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:34:58.245752 master-1 kubenswrapper[4771]: I1011 10:34:58.245707 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:02.990859 master-1 kubenswrapper[4771]: I1011 10:35:02.990702 4771 generic.go:334] "Generic (PLEG): container finished" podID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerID="2fd6e0cb14ecdcadbf2571f6d4dd1d2a4a1e6cf999fc333d09b9fc98b284b780" exitCode=0 Oct 11 10:35:02.990859 master-1 kubenswrapper[4771]: I1011 10:35:02.990768 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" event={"ID":"04cd4a19-2532-43d1-9144-1f59d9e52d19","Type":"ContainerDied","Data":"2fd6e0cb14ecdcadbf2571f6d4dd1d2a4a1e6cf999fc333d09b9fc98b284b780"} Oct 11 10:35:02.990859 master-1 kubenswrapper[4771]: I1011 10:35:02.990853 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" event={"ID":"04cd4a19-2532-43d1-9144-1f59d9e52d19","Type":"ContainerStarted","Data":"143e04eafcd2b03e93df12dc4cef70c9cbf812c2f07ee907f7529b8a34ff8d77"} Oct 11 10:35:02.990859 master-1 kubenswrapper[4771]: I1011 10:35:02.990885 4771 scope.go:117] "RemoveContainer" containerID="d9d09acfb9b74efc71914e418c9f7ad84873a3a13515d6cfcddf159cfd555604" Oct 11 10:35:03.069712 master-1 kubenswrapper[4771]: E1011 10:35:03.069576 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" podUID="537a2b50-0394-47bd-941a-def350316943" Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: I1011 10:35:03.244974 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:03.245143 master-1 kubenswrapper[4771]: I1011 10:35:03.245075 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:03.494799 master-1 kubenswrapper[4771]: I1011 10:35:03.494690 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:35:03.499570 master-1 kubenswrapper[4771]: I1011 10:35:03.499408 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:03.499570 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:03.499570 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:03.499570 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:03.499869 master-1 kubenswrapper[4771]: I1011 10:35:03.499560 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:04.000665 master-1 kubenswrapper[4771]: I1011 10:35:04.000575 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:35:04.324250 master-1 kubenswrapper[4771]: I1011 10:35:04.324127 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") pod \"route-controller-manager-5bcc5987f5-f92xw\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:35:04.324596 master-1 kubenswrapper[4771]: E1011 10:35:04.324348 4771 configmap.go:193] Couldn't get configMap openshift-route-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:35:04.324712 master-1 kubenswrapper[4771]: E1011 10:35:04.324632 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca podName:537a2b50-0394-47bd-941a-def350316943 nodeName:}" failed. No retries permitted until 2025-10-11 10:37:06.324585174 +0000 UTC m=+658.298811645 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca") pod "route-controller-manager-5bcc5987f5-f92xw" (UID: "537a2b50-0394-47bd-941a-def350316943") : configmap "client-ca" not found Oct 11 10:35:04.496881 master-1 kubenswrapper[4771]: I1011 10:35:04.496742 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:04.496881 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:04.496881 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:04.496881 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:04.497503 master-1 kubenswrapper[4771]: I1011 10:35:04.496911 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:05.083595 master-1 kubenswrapper[4771]: E1011 10:35:05.083492 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" podUID="c9e9455e-0b47-4623-9b4c-ef79cf62a254" Oct 11 10:35:05.494649 master-1 kubenswrapper[4771]: I1011 10:35:05.494471 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:35:05.497596 master-1 kubenswrapper[4771]: I1011 10:35:05.497531 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:05.497596 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:05.497596 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:05.497596 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:05.497739 master-1 kubenswrapper[4771]: I1011 10:35:05.497623 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:06.011059 master-1 kubenswrapper[4771]: I1011 10:35:06.010964 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:35:06.497587 master-1 kubenswrapper[4771]: I1011 10:35:06.497521 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:06.497587 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:06.497587 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:06.497587 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:06.498591 master-1 kubenswrapper[4771]: I1011 10:35:06.498509 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:06.761620 master-1 kubenswrapper[4771]: I1011 10:35:06.760735 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") pod \"controller-manager-565f857764-nhm4g\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:35:06.761620 master-1 kubenswrapper[4771]: E1011 10:35:06.760964 4771 configmap.go:193] Couldn't get configMap openshift-controller-manager/client-ca: configmap "client-ca" not found Oct 11 10:35:06.761620 master-1 kubenswrapper[4771]: E1011 10:35:06.761118 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca podName:c9e9455e-0b47-4623-9b4c-ef79cf62a254 nodeName:}" failed. No retries permitted until 2025-10-11 10:37:08.761087995 +0000 UTC m=+660.735314436 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "client-ca" (UniqueName: "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca") pod "controller-manager-565f857764-nhm4g" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254") : configmap "client-ca" not found Oct 11 10:35:07.498311 master-1 kubenswrapper[4771]: I1011 10:35:07.498205 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:07.498311 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:07.498311 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:07.498311 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:07.499242 master-1 kubenswrapper[4771]: I1011 10:35:07.498319 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: I1011 10:35:08.242542 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:08.242620 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:08.245553 master-1 kubenswrapper[4771]: I1011 10:35:08.242647 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:08.245553 master-1 kubenswrapper[4771]: I1011 10:35:08.242779 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: I1011 10:35:08.248836 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:08.249016 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:08.251430 master-1 kubenswrapper[4771]: I1011 10:35:08.249032 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:08.498303 master-1 kubenswrapper[4771]: I1011 10:35:08.498107 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:08.498303 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:08.498303 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:08.498303 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:08.498303 master-1 kubenswrapper[4771]: I1011 10:35:08.498189 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:09.497126 master-1 kubenswrapper[4771]: I1011 10:35:09.497023 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:09.497126 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:09.497126 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:09.497126 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:09.497672 master-1 kubenswrapper[4771]: I1011 10:35:09.497142 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: I1011 10:35:10.248306 4771 patch_prober.go:28] interesting pod/metrics-server-65d86dff78-bg7lk container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: [+]metric-storage-ready ok Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: [+]metric-informer-sync ok Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: [+]metadata-informer-sync ok Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:10.248461 master-1 kubenswrapper[4771]: I1011 10:35:10.248456 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:10.497480 master-1 kubenswrapper[4771]: I1011 10:35:10.497327 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:10.497480 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:10.497480 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:10.497480 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:10.498114 master-1 kubenswrapper[4771]: I1011 10:35:10.497519 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:11.497726 master-1 kubenswrapper[4771]: I1011 10:35:11.497607 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:11.497726 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:11.497726 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:11.497726 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:11.497726 master-1 kubenswrapper[4771]: I1011 10:35:11.497724 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:12.498624 master-1 kubenswrapper[4771]: I1011 10:35:12.498530 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:12.498624 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:12.498624 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:12.498624 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:12.499826 master-1 kubenswrapper[4771]: I1011 10:35:12.499642 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: I1011 10:35:13.244567 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:13.244663 master-1 kubenswrapper[4771]: I1011 10:35:13.244656 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:13.497840 master-1 kubenswrapper[4771]: I1011 10:35:13.497655 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:13.497840 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:13.497840 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:13.497840 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:13.497840 master-1 kubenswrapper[4771]: I1011 10:35:13.497746 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:14.498075 master-1 kubenswrapper[4771]: I1011 10:35:14.497968 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:14.498075 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:14.498075 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:14.498075 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:14.498075 master-1 kubenswrapper[4771]: I1011 10:35:14.498070 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:15.498297 master-1 kubenswrapper[4771]: I1011 10:35:15.498197 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:15.498297 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:15.498297 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:15.498297 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:15.498297 master-1 kubenswrapper[4771]: I1011 10:35:15.498286 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:16.497269 master-1 kubenswrapper[4771]: I1011 10:35:16.497118 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:16.497269 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:16.497269 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:16.497269 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:16.497811 master-1 kubenswrapper[4771]: I1011 10:35:16.497267 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:17.497709 master-1 kubenswrapper[4771]: I1011 10:35:17.497621 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:17.497709 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:17.497709 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:17.497709 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:17.497709 master-1 kubenswrapper[4771]: I1011 10:35:17.497713 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:17.791638 master-1 kubenswrapper[4771]: I1011 10:35:17.791449 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: I1011 10:35:18.245748 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:18.245842 master-1 kubenswrapper[4771]: I1011 10:35:18.245842 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:18.497836 master-1 kubenswrapper[4771]: I1011 10:35:18.497579 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:18.497836 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:18.497836 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:18.497836 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:18.497836 master-1 kubenswrapper[4771]: I1011 10:35:18.497697 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:19.497404 master-1 kubenswrapper[4771]: I1011 10:35:19.497285 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:19.497404 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:19.497404 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:19.497404 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:19.498629 master-1 kubenswrapper[4771]: I1011 10:35:19.497437 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:20.497239 master-1 kubenswrapper[4771]: I1011 10:35:20.497154 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:20.497239 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:20.497239 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:20.497239 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:20.498009 master-1 kubenswrapper[4771]: I1011 10:35:20.497245 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:21.497279 master-1 kubenswrapper[4771]: I1011 10:35:21.497174 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:21.497279 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:21.497279 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:21.497279 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:21.497694 master-1 kubenswrapper[4771]: I1011 10:35:21.497277 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:22.497468 master-1 kubenswrapper[4771]: I1011 10:35:22.497399 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:22.497468 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:22.497468 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:22.497468 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:22.498676 master-1 kubenswrapper[4771]: I1011 10:35:22.497480 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: I1011 10:35:23.245998 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:23.246096 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:23.248590 master-1 kubenswrapper[4771]: I1011 10:35:23.246113 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:23.498346 master-1 kubenswrapper[4771]: I1011 10:35:23.498210 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:23.498346 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:23.498346 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:23.498346 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:23.499397 master-1 kubenswrapper[4771]: I1011 10:35:23.499324 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:24.497091 master-1 kubenswrapper[4771]: I1011 10:35:24.497012 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:24.497091 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:24.497091 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:24.497091 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:24.497613 master-1 kubenswrapper[4771]: I1011 10:35:24.497134 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:25.498890 master-1 kubenswrapper[4771]: I1011 10:35:25.498747 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:25.498890 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:25.498890 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:25.498890 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:25.499914 master-1 kubenswrapper[4771]: I1011 10:35:25.498962 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:26.496974 master-1 kubenswrapper[4771]: I1011 10:35:26.496875 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:26.496974 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:26.496974 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:26.496974 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:26.496974 master-1 kubenswrapper[4771]: I1011 10:35:26.496969 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:27.497412 master-1 kubenswrapper[4771]: I1011 10:35:27.497290 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:27.497412 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:27.497412 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:27.497412 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:27.497412 master-1 kubenswrapper[4771]: I1011 10:35:27.497379 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:28.041998 master-1 kubenswrapper[4771]: I1011 10:35:28.041900 4771 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: I1011 10:35:28.245953 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:28.246067 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:28.248417 master-1 kubenswrapper[4771]: I1011 10:35:28.246073 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:28.497230 master-1 kubenswrapper[4771]: I1011 10:35:28.497131 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:28.497230 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:28.497230 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:28.497230 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:28.498296 master-1 kubenswrapper[4771]: I1011 10:35:28.497234 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:29.497433 master-1 kubenswrapper[4771]: I1011 10:35:29.497312 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:29.497433 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:29.497433 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:29.497433 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:29.497433 master-1 kubenswrapper[4771]: I1011 10:35:29.497435 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: I1011 10:35:30.245903 4771 patch_prober.go:28] interesting pod/metrics-server-65d86dff78-bg7lk container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: [+]metric-storage-ready ok Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: [+]metric-informer-sync ok Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: [+]metadata-informer-sync ok Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:30.245992 master-1 kubenswrapper[4771]: I1011 10:35:30.245988 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:30.497413 master-1 kubenswrapper[4771]: I1011 10:35:30.497208 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:30.497413 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:30.497413 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:30.497413 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:30.497413 master-1 kubenswrapper[4771]: I1011 10:35:30.497279 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:31.497944 master-1 kubenswrapper[4771]: I1011 10:35:31.497852 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:31.497944 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:31.497944 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:31.497944 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:31.498972 master-1 kubenswrapper[4771]: I1011 10:35:31.497955 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:32.498135 master-1 kubenswrapper[4771]: I1011 10:35:32.498057 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:32.498135 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:32.498135 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:32.498135 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:32.499427 master-1 kubenswrapper[4771]: I1011 10:35:32.498147 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: I1011 10:35:33.245782 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:33.245865 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:33.248158 master-1 kubenswrapper[4771]: I1011 10:35:33.245892 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:33.498055 master-1 kubenswrapper[4771]: I1011 10:35:33.497893 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:33.498055 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:33.498055 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:33.498055 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:33.498055 master-1 kubenswrapper[4771]: I1011 10:35:33.497988 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:34.497256 master-1 kubenswrapper[4771]: I1011 10:35:34.497160 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:34.497256 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:34.497256 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:34.497256 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:34.497256 master-1 kubenswrapper[4771]: I1011 10:35:34.497252 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:35.497382 master-1 kubenswrapper[4771]: I1011 10:35:35.497297 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:35.497382 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:35.497382 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:35.497382 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:35.498341 master-1 kubenswrapper[4771]: I1011 10:35:35.497483 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:36.497972 master-1 kubenswrapper[4771]: I1011 10:35:36.497869 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:36.497972 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:36.497972 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:36.497972 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:36.498932 master-1 kubenswrapper[4771]: I1011 10:35:36.498005 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:37.498605 master-1 kubenswrapper[4771]: I1011 10:35:37.498546 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:37.498605 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:37.498605 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:37.498605 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:37.499443 master-1 kubenswrapper[4771]: I1011 10:35:37.499411 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: I1011 10:35:38.244848 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:38.244921 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:38.247757 master-1 kubenswrapper[4771]: I1011 10:35:38.245591 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:38.497512 master-1 kubenswrapper[4771]: I1011 10:35:38.497240 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:38.497512 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:38.497512 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:38.497512 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:38.497512 master-1 kubenswrapper[4771]: I1011 10:35:38.497381 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:39.498111 master-1 kubenswrapper[4771]: I1011 10:35:39.498042 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:39.498111 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:39.498111 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:39.498111 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:39.498711 master-1 kubenswrapper[4771]: I1011 10:35:39.498128 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:39.731465 master-1 kubenswrapper[4771]: I1011 10:35:39.731410 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-565f857764-nhm4g"] Oct 11 10:35:39.731852 master-1 kubenswrapper[4771]: E1011 10:35:39.731818 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" podUID="c9e9455e-0b47-4623-9b4c-ef79cf62a254" Oct 11 10:35:39.746910 master-1 kubenswrapper[4771]: I1011 10:35:39.746848 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw"] Oct 11 10:35:39.747825 master-1 kubenswrapper[4771]: E1011 10:35:39.747792 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[client-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" podUID="537a2b50-0394-47bd-941a-def350316943" Oct 11 10:35:40.237218 master-1 kubenswrapper[4771]: I1011 10:35:40.237144 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:35:40.237774 master-1 kubenswrapper[4771]: I1011 10:35:40.237173 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:35:40.249968 master-1 kubenswrapper[4771]: I1011 10:35:40.249902 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:35:40.257953 master-1 kubenswrapper[4771]: I1011 10:35:40.257900 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:35:40.350538 master-1 kubenswrapper[4771]: I1011 10:35:40.350456 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/537a2b50-0394-47bd-941a-def350316943-serving-cert\") pod \"537a2b50-0394-47bd-941a-def350316943\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " Oct 11 10:35:40.350827 master-1 kubenswrapper[4771]: I1011 10:35:40.350543 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-config\") pod \"537a2b50-0394-47bd-941a-def350316943\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " Oct 11 10:35:40.350827 master-1 kubenswrapper[4771]: I1011 10:35:40.350617 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wp47\" (UniqueName: \"kubernetes.io/projected/c9e9455e-0b47-4623-9b4c-ef79cf62a254-kube-api-access-9wp47\") pod \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " Oct 11 10:35:40.350827 master-1 kubenswrapper[4771]: I1011 10:35:40.350666 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9e9455e-0b47-4623-9b4c-ef79cf62a254-serving-cert\") pod \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " Oct 11 10:35:40.350827 master-1 kubenswrapper[4771]: I1011 10:35:40.350713 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-config\") pod \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " Oct 11 10:35:40.350827 master-1 kubenswrapper[4771]: I1011 10:35:40.350753 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-proxy-ca-bundles\") pod \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\" (UID: \"c9e9455e-0b47-4623-9b4c-ef79cf62a254\") " Oct 11 10:35:40.351383 master-1 kubenswrapper[4771]: I1011 10:35:40.351302 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwxw7\" (UniqueName: \"kubernetes.io/projected/537a2b50-0394-47bd-941a-def350316943-kube-api-access-zwxw7\") pod \"537a2b50-0394-47bd-941a-def350316943\" (UID: \"537a2b50-0394-47bd-941a-def350316943\") " Oct 11 10:35:40.351654 master-1 kubenswrapper[4771]: I1011 10:35:40.351596 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-proxy-ca-bundles" (OuterVolumeSpecName: "proxy-ca-bundles") pod "c9e9455e-0b47-4623-9b4c-ef79cf62a254" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254"). InnerVolumeSpecName "proxy-ca-bundles". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:35:40.351787 master-1 kubenswrapper[4771]: I1011 10:35:40.351751 4771 reconciler_common.go:293] "Volume detached for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-proxy-ca-bundles\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:40.352045 master-1 kubenswrapper[4771]: I1011 10:35:40.351975 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-config" (OuterVolumeSpecName: "config") pod "537a2b50-0394-47bd-941a-def350316943" (UID: "537a2b50-0394-47bd-941a-def350316943"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:35:40.352580 master-1 kubenswrapper[4771]: I1011 10:35:40.352478 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-config" (OuterVolumeSpecName: "config") pod "c9e9455e-0b47-4623-9b4c-ef79cf62a254" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:35:40.358349 master-1 kubenswrapper[4771]: I1011 10:35:40.358252 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537a2b50-0394-47bd-941a-def350316943-kube-api-access-zwxw7" (OuterVolumeSpecName: "kube-api-access-zwxw7") pod "537a2b50-0394-47bd-941a-def350316943" (UID: "537a2b50-0394-47bd-941a-def350316943"). InnerVolumeSpecName "kube-api-access-zwxw7". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:35:40.359852 master-1 kubenswrapper[4771]: I1011 10:35:40.359758 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e9455e-0b47-4623-9b4c-ef79cf62a254-kube-api-access-9wp47" (OuterVolumeSpecName: "kube-api-access-9wp47") pod "c9e9455e-0b47-4623-9b4c-ef79cf62a254" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254"). InnerVolumeSpecName "kube-api-access-9wp47". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:35:40.359852 master-1 kubenswrapper[4771]: I1011 10:35:40.359811 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537a2b50-0394-47bd-941a-def350316943-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "537a2b50-0394-47bd-941a-def350316943" (UID: "537a2b50-0394-47bd-941a-def350316943"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:35:40.360185 master-1 kubenswrapper[4771]: I1011 10:35:40.360095 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e9455e-0b47-4623-9b4c-ef79cf62a254-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "c9e9455e-0b47-4623-9b4c-ef79cf62a254" (UID: "c9e9455e-0b47-4623-9b4c-ef79cf62a254"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:35:40.452755 master-1 kubenswrapper[4771]: I1011 10:35:40.452723 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/537a2b50-0394-47bd-941a-def350316943-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:40.452894 master-1 kubenswrapper[4771]: I1011 10:35:40.452880 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:40.452990 master-1 kubenswrapper[4771]: I1011 10:35:40.452975 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-9wp47\" (UniqueName: \"kubernetes.io/projected/c9e9455e-0b47-4623-9b4c-ef79cf62a254-kube-api-access-9wp47\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:40.453066 master-1 kubenswrapper[4771]: I1011 10:35:40.453054 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9e9455e-0b47-4623-9b4c-ef79cf62a254-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:40.453145 master-1 kubenswrapper[4771]: I1011 10:35:40.453133 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:40.453223 master-1 kubenswrapper[4771]: I1011 10:35:40.453210 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-zwxw7\" (UniqueName: \"kubernetes.io/projected/537a2b50-0394-47bd-941a-def350316943-kube-api-access-zwxw7\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:40.496094 master-1 kubenswrapper[4771]: I1011 10:35:40.495935 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:40.496094 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:40.496094 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:40.496094 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:40.496702 master-1 kubenswrapper[4771]: I1011 10:35:40.496659 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:41.242989 master-1 kubenswrapper[4771]: I1011 10:35:41.242920 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-565f857764-nhm4g" Oct 11 10:35:41.243991 master-1 kubenswrapper[4771]: I1011 10:35:41.243584 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw" Oct 11 10:35:41.293135 master-1 kubenswrapper[4771]: I1011 10:35:41.293041 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-565f857764-nhm4g"] Oct 11 10:35:41.298551 master-1 kubenswrapper[4771]: I1011 10:35:41.298478 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-controller-manager/controller-manager-565f857764-nhm4g"] Oct 11 10:35:41.333160 master-1 kubenswrapper[4771]: I1011 10:35:41.333022 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw"] Oct 11 10:35:41.339585 master-1 kubenswrapper[4771]: I1011 10:35:41.339520 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-5bcc5987f5-f92xw"] Oct 11 10:35:41.365321 master-1 kubenswrapper[4771]: I1011 10:35:41.365229 4771 reconciler_common.go:293] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/537a2b50-0394-47bd-941a-def350316943-client-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:41.365321 master-1 kubenswrapper[4771]: I1011 10:35:41.365291 4771 reconciler_common.go:293] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c9e9455e-0b47-4623-9b4c-ef79cf62a254-client-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:41.498349 master-1 kubenswrapper[4771]: I1011 10:35:41.498167 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:41.498349 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:41.498349 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:41.498349 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:41.498349 master-1 kubenswrapper[4771]: I1011 10:35:41.498256 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:42.446053 master-1 kubenswrapper[4771]: I1011 10:35:42.445958 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="537a2b50-0394-47bd-941a-def350316943" path="/var/lib/kubelet/pods/537a2b50-0394-47bd-941a-def350316943/volumes" Oct 11 10:35:42.446908 master-1 kubenswrapper[4771]: I1011 10:35:42.446730 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9e9455e-0b47-4623-9b4c-ef79cf62a254" path="/var/lib/kubelet/pods/c9e9455e-0b47-4623-9b4c-ef79cf62a254/volumes" Oct 11 10:35:42.497527 master-1 kubenswrapper[4771]: I1011 10:35:42.497466 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:42.497527 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:42.497527 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:42.497527 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:42.498153 master-1 kubenswrapper[4771]: I1011 10:35:42.498111 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: I1011 10:35:43.247638 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:43.247731 master-1 kubenswrapper[4771]: I1011 10:35:43.247727 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:43.496997 master-1 kubenswrapper[4771]: I1011 10:35:43.496917 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:43.496997 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:43.496997 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:43.496997 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:43.497997 master-1 kubenswrapper[4771]: I1011 10:35:43.497020 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:43.940896 master-1 kubenswrapper[4771]: I1011 10:35:43.940821 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z"] Oct 11 10:35:43.941541 master-1 kubenswrapper[4771]: E1011 10:35:43.941058 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="c1c3b2b9-8880-496b-88ed-9706cd8ee23d" containerName="installer" Oct 11 10:35:43.941541 master-1 kubenswrapper[4771]: I1011 10:35:43.941074 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c3b2b9-8880-496b-88ed-9706cd8ee23d" containerName="installer" Oct 11 10:35:43.941541 master-1 kubenswrapper[4771]: I1011 10:35:43.941179 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="c1c3b2b9-8880-496b-88ed-9706cd8ee23d" containerName="installer" Oct 11 10:35:43.941944 master-1 kubenswrapper[4771]: I1011 10:35:43.941619 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:43.945528 master-1 kubenswrapper[4771]: I1011 10:35:43.945479 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"config" Oct 11 10:35:43.945741 master-1 kubenswrapper[4771]: I1011 10:35:43.945556 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-controller-manager"/"serving-cert" Oct 11 10:35:43.945837 master-1 kubenswrapper[4771]: I1011 10:35:43.945761 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-service-ca.crt" Oct 11 10:35:43.946186 master-1 kubenswrapper[4771]: I1011 10:35:43.945907 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"kube-root-ca.crt" Oct 11 10:35:43.948393 master-1 kubenswrapper[4771]: I1011 10:35:43.948328 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"client-ca" Oct 11 10:35:43.950005 master-1 kubenswrapper[4771]: I1011 10:35:43.949954 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-global-ca" Oct 11 10:35:43.950238 master-1 kubenswrapper[4771]: I1011 10:35:43.950116 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m"] Oct 11 10:35:43.951922 master-1 kubenswrapper[4771]: I1011 10:35:43.951879 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:43.956448 master-1 kubenswrapper[4771]: I1011 10:35:43.955187 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-route-controller-manager"/"serving-cert" Oct 11 10:35:43.959554 master-1 kubenswrapper[4771]: I1011 10:35:43.958760 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z"] Oct 11 10:35:43.959554 master-1 kubenswrapper[4771]: I1011 10:35:43.959515 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"client-ca" Oct 11 10:35:43.960001 master-1 kubenswrapper[4771]: I1011 10:35:43.959587 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"config" Oct 11 10:35:43.960001 master-1 kubenswrapper[4771]: I1011 10:35:43.959594 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"openshift-service-ca.crt" Oct 11 10:35:43.960001 master-1 kubenswrapper[4771]: I1011 10:35:43.959702 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"kube-root-ca.crt" Oct 11 10:35:43.964726 master-1 kubenswrapper[4771]: I1011 10:35:43.964669 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m"] Oct 11 10:35:44.003018 master-1 kubenswrapper[4771]: I1011 10:35:44.002961 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62f55273-2711-4de4-a399-5dae9b578f0c-serving-cert\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.003018 master-1 kubenswrapper[4771]: I1011 10:35:44.003014 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-config\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.003251 master-1 kubenswrapper[4771]: I1011 10:35:44.003051 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-config\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.003251 master-1 kubenswrapper[4771]: I1011 10:35:44.003138 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-client-ca\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.003251 master-1 kubenswrapper[4771]: I1011 10:35:44.003181 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-client-ca\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.003251 master-1 kubenswrapper[4771]: I1011 10:35:44.003223 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqtc\" (UniqueName: \"kubernetes.io/projected/62f55273-2711-4de4-a399-5dae9b578f0c-kube-api-access-thqtc\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.003651 master-1 kubenswrapper[4771]: I1011 10:35:44.003603 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zmdm\" (UniqueName: \"kubernetes.io/projected/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-kube-api-access-9zmdm\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.003779 master-1 kubenswrapper[4771]: I1011 10:35:44.003697 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-serving-cert\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.003848 master-1 kubenswrapper[4771]: I1011 10:35:44.003811 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-proxy-ca-bundles\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.104628 master-1 kubenswrapper[4771]: I1011 10:35:44.104507 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-proxy-ca-bundles\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.104628 master-1 kubenswrapper[4771]: I1011 10:35:44.104577 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62f55273-2711-4de4-a399-5dae9b578f0c-serving-cert\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.104628 master-1 kubenswrapper[4771]: I1011 10:35:44.104605 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-config\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.104628 master-1 kubenswrapper[4771]: I1011 10:35:44.104634 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-config\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.105182 master-1 kubenswrapper[4771]: I1011 10:35:44.104674 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-client-ca\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.105182 master-1 kubenswrapper[4771]: I1011 10:35:44.104711 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-client-ca\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.105182 master-1 kubenswrapper[4771]: I1011 10:35:44.104751 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-thqtc\" (UniqueName: \"kubernetes.io/projected/62f55273-2711-4de4-a399-5dae9b578f0c-kube-api-access-thqtc\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.105182 master-1 kubenswrapper[4771]: I1011 10:35:44.104791 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-9zmdm\" (UniqueName: \"kubernetes.io/projected/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-kube-api-access-9zmdm\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.105182 master-1 kubenswrapper[4771]: I1011 10:35:44.104813 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-serving-cert\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.106798 master-1 kubenswrapper[4771]: I1011 10:35:44.106703 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-client-ca\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.106798 master-1 kubenswrapper[4771]: I1011 10:35:44.106771 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-client-ca\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.107777 master-1 kubenswrapper[4771]: I1011 10:35:44.107518 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-proxy-ca-bundles\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.108335 master-1 kubenswrapper[4771]: I1011 10:35:44.107877 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-config\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.108335 master-1 kubenswrapper[4771]: I1011 10:35:44.108161 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-config\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.110067 master-1 kubenswrapper[4771]: I1011 10:35:44.110000 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62f55273-2711-4de4-a399-5dae9b578f0c-serving-cert\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.110067 master-1 kubenswrapper[4771]: I1011 10:35:44.110060 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-serving-cert\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.124962 master-1 kubenswrapper[4771]: I1011 10:35:44.124914 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zmdm\" (UniqueName: \"kubernetes.io/projected/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-kube-api-access-9zmdm\") pod \"route-controller-manager-68b68f45cd-mqn2m\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.125929 master-1 kubenswrapper[4771]: I1011 10:35:44.125889 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqtc\" (UniqueName: \"kubernetes.io/projected/62f55273-2711-4de4-a399-5dae9b578f0c-kube-api-access-thqtc\") pod \"controller-manager-5cf7cfc4c5-6jg5z\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.266770 master-1 kubenswrapper[4771]: I1011 10:35:44.266595 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:44.291344 master-1 kubenswrapper[4771]: I1011 10:35:44.291219 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:44.497821 master-1 kubenswrapper[4771]: I1011 10:35:44.496793 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:44.497821 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:44.497821 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:44.497821 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:44.497821 master-1 kubenswrapper[4771]: I1011 10:35:44.496860 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:44.583009 master-1 kubenswrapper[4771]: I1011 10:35:44.582736 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m"] Oct 11 10:35:44.586999 master-1 kubenswrapper[4771]: W1011 10:35:44.586934 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89d5fa2_4b2d_47b8_9f43_fbf5942eaff3.slice/crio-fb8e606f605b5a7a5119eb59ac6c30ff451c4fbab3f45cf0454534a92053916c WatchSource:0}: Error finding container fb8e606f605b5a7a5119eb59ac6c30ff451c4fbab3f45cf0454534a92053916c: Status 404 returned error can't find the container with id fb8e606f605b5a7a5119eb59ac6c30ff451c4fbab3f45cf0454534a92053916c Oct 11 10:35:44.721761 master-1 kubenswrapper[4771]: I1011 10:35:44.721651 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z"] Oct 11 10:35:44.730878 master-1 kubenswrapper[4771]: W1011 10:35:44.730817 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f55273_2711_4de4_a399_5dae9b578f0c.slice/crio-b1d9019c57f2da10f531e4404f9aaba506a3aa469ba7140f3ac23a4a053a6c4e WatchSource:0}: Error finding container b1d9019c57f2da10f531e4404f9aaba506a3aa469ba7140f3ac23a4a053a6c4e: Status 404 returned error can't find the container with id b1d9019c57f2da10f531e4404f9aaba506a3aa469ba7140f3ac23a4a053a6c4e Oct 11 10:35:45.271861 master-1 kubenswrapper[4771]: I1011 10:35:45.271786 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" event={"ID":"62f55273-2711-4de4-a399-5dae9b578f0c","Type":"ContainerStarted","Data":"b1d9019c57f2da10f531e4404f9aaba506a3aa469ba7140f3ac23a4a053a6c4e"} Oct 11 10:35:45.274198 master-1 kubenswrapper[4771]: I1011 10:35:45.274147 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" event={"ID":"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3","Type":"ContainerStarted","Data":"fb8e606f605b5a7a5119eb59ac6c30ff451c4fbab3f45cf0454534a92053916c"} Oct 11 10:35:45.497066 master-1 kubenswrapper[4771]: I1011 10:35:45.497002 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:45.497066 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:45.497066 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:45.497066 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:45.497066 master-1 kubenswrapper[4771]: I1011 10:35:45.497084 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:46.497294 master-1 kubenswrapper[4771]: I1011 10:35:46.497210 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:46.497294 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:46.497294 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:46.497294 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:46.498061 master-1 kubenswrapper[4771]: I1011 10:35:46.497299 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:47.496730 master-1 kubenswrapper[4771]: I1011 10:35:47.496665 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:47.496730 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:47.496730 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:47.496730 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:47.497056 master-1 kubenswrapper[4771]: I1011 10:35:47.496758 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:47.794692 master-1 kubenswrapper[4771]: I1011 10:35:47.790171 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_machine-config-server-h7gnk_6d20faa4-e5eb-4766-b4f5-30e491d1820c/machine-config-server/0.log" Oct 11 10:35:48.113761 master-1 kubenswrapper[4771]: I1011 10:35:48.112279 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z"] Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: I1011 10:35:48.242306 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:48.242384 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:48.243845 master-1 kubenswrapper[4771]: I1011 10:35:48.242400 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:48.296207 master-1 kubenswrapper[4771]: I1011 10:35:48.296143 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" event={"ID":"62f55273-2711-4de4-a399-5dae9b578f0c","Type":"ContainerStarted","Data":"3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3"} Oct 11 10:35:48.296468 master-1 kubenswrapper[4771]: I1011 10:35:48.296227 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" podUID="62f55273-2711-4de4-a399-5dae9b578f0c" containerName="controller-manager" containerID="cri-o://3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3" gracePeriod=30 Oct 11 10:35:48.296468 master-1 kubenswrapper[4771]: I1011 10:35:48.296386 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:48.297958 master-1 kubenswrapper[4771]: I1011 10:35:48.297807 4771 patch_prober.go:28] interesting pod/controller-manager-5cf7cfc4c5-6jg5z container/controller-manager namespace/openshift-controller-manager: Readiness probe status=failure output="Get \"https://10.129.0.54:8443/healthz\": dial tcp 10.129.0.54:8443: connect: connection refused" start-of-body= Oct 11 10:35:48.297958 master-1 kubenswrapper[4771]: I1011 10:35:48.297874 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" podUID="62f55273-2711-4de4-a399-5dae9b578f0c" containerName="controller-manager" probeResult="failure" output="Get \"https://10.129.0.54:8443/healthz\": dial tcp 10.129.0.54:8443: connect: connection refused" Oct 11 10:35:48.298213 master-1 kubenswrapper[4771]: I1011 10:35:48.298151 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" event={"ID":"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3","Type":"ContainerStarted","Data":"5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547"} Oct 11 10:35:48.298603 master-1 kubenswrapper[4771]: I1011 10:35:48.298553 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:48.300092 master-1 kubenswrapper[4771]: I1011 10:35:48.300044 4771 patch_prober.go:28] interesting pod/route-controller-manager-68b68f45cd-mqn2m container/route-controller-manager namespace/openshift-route-controller-manager: Readiness probe status=failure output="Get \"https://10.129.0.55:8443/healthz\": dial tcp 10.129.0.55:8443: connect: connection refused" start-of-body= Oct 11 10:35:48.300165 master-1 kubenswrapper[4771]: I1011 10:35:48.300098 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" podUID="e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" containerName="route-controller-manager" probeResult="failure" output="Get \"https://10.129.0.55:8443/healthz\": dial tcp 10.129.0.55:8443: connect: connection refused" Oct 11 10:35:48.317939 master-1 kubenswrapper[4771]: I1011 10:35:48.317841 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" podStartSLOduration=5.923964217 podStartE2EDuration="9.317813347s" podCreationTimestamp="2025-10-11 10:35:39 +0000 UTC" firstStartedPulling="2025-10-11 10:35:44.733467834 +0000 UTC m=+576.707694285" lastFinishedPulling="2025-10-11 10:35:48.127316974 +0000 UTC m=+580.101543415" observedRunningTime="2025-10-11 10:35:48.31655916 +0000 UTC m=+580.290785591" watchObservedRunningTime="2025-10-11 10:35:48.317813347 +0000 UTC m=+580.292039828" Oct 11 10:35:48.343137 master-1 kubenswrapper[4771]: I1011 10:35:48.343043 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" podStartSLOduration=5.805813347 podStartE2EDuration="9.343024417s" podCreationTimestamp="2025-10-11 10:35:39 +0000 UTC" firstStartedPulling="2025-10-11 10:35:44.590488035 +0000 UTC m=+576.564714486" lastFinishedPulling="2025-10-11 10:35:48.127699105 +0000 UTC m=+580.101925556" observedRunningTime="2025-10-11 10:35:48.341260205 +0000 UTC m=+580.315486696" watchObservedRunningTime="2025-10-11 10:35:48.343024417 +0000 UTC m=+580.317250848" Oct 11 10:35:48.497947 master-1 kubenswrapper[4771]: I1011 10:35:48.497891 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:48.497947 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:48.497947 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:48.497947 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:48.498241 master-1 kubenswrapper[4771]: I1011 10:35:48.497982 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:48.681048 master-1 kubenswrapper[4771]: I1011 10:35:48.680688 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-controller-manager_controller-manager-5cf7cfc4c5-6jg5z_62f55273-2711-4de4-a399-5dae9b578f0c/controller-manager/0.log" Oct 11 10:35:48.681048 master-1 kubenswrapper[4771]: I1011 10:35:48.680790 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:48.691240 master-1 kubenswrapper[4771]: I1011 10:35:48.691190 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-config\") pod \"62f55273-2711-4de4-a399-5dae9b578f0c\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " Oct 11 10:35:48.691240 master-1 kubenswrapper[4771]: I1011 10:35:48.691239 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62f55273-2711-4de4-a399-5dae9b578f0c-serving-cert\") pod \"62f55273-2711-4de4-a399-5dae9b578f0c\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " Oct 11 10:35:48.691409 master-1 kubenswrapper[4771]: I1011 10:35:48.691288 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thqtc\" (UniqueName: \"kubernetes.io/projected/62f55273-2711-4de4-a399-5dae9b578f0c-kube-api-access-thqtc\") pod \"62f55273-2711-4de4-a399-5dae9b578f0c\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " Oct 11 10:35:48.691409 master-1 kubenswrapper[4771]: I1011 10:35:48.691316 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-proxy-ca-bundles\") pod \"62f55273-2711-4de4-a399-5dae9b578f0c\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " Oct 11 10:35:48.691409 master-1 kubenswrapper[4771]: I1011 10:35:48.691339 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-client-ca\") pod \"62f55273-2711-4de4-a399-5dae9b578f0c\" (UID: \"62f55273-2711-4de4-a399-5dae9b578f0c\") " Oct 11 10:35:48.692300 master-1 kubenswrapper[4771]: I1011 10:35:48.692273 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-client-ca" (OuterVolumeSpecName: "client-ca") pod "62f55273-2711-4de4-a399-5dae9b578f0c" (UID: "62f55273-2711-4de4-a399-5dae9b578f0c"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:35:48.692802 master-1 kubenswrapper[4771]: I1011 10:35:48.692448 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-config" (OuterVolumeSpecName: "config") pod "62f55273-2711-4de4-a399-5dae9b578f0c" (UID: "62f55273-2711-4de4-a399-5dae9b578f0c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:35:48.692802 master-1 kubenswrapper[4771]: I1011 10:35:48.692625 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-proxy-ca-bundles" (OuterVolumeSpecName: "proxy-ca-bundles") pod "62f55273-2711-4de4-a399-5dae9b578f0c" (UID: "62f55273-2711-4de4-a399-5dae9b578f0c"). InnerVolumeSpecName "proxy-ca-bundles". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:35:48.696417 master-1 kubenswrapper[4771]: I1011 10:35:48.696339 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f55273-2711-4de4-a399-5dae9b578f0c-kube-api-access-thqtc" (OuterVolumeSpecName: "kube-api-access-thqtc") pod "62f55273-2711-4de4-a399-5dae9b578f0c" (UID: "62f55273-2711-4de4-a399-5dae9b578f0c"). InnerVolumeSpecName "kube-api-access-thqtc". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:35:48.699878 master-1 kubenswrapper[4771]: I1011 10:35:48.699806 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f55273-2711-4de4-a399-5dae9b578f0c-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "62f55273-2711-4de4-a399-5dae9b578f0c" (UID: "62f55273-2711-4de4-a399-5dae9b578f0c"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:35:48.792758 master-1 kubenswrapper[4771]: I1011 10:35:48.792657 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-thqtc\" (UniqueName: \"kubernetes.io/projected/62f55273-2711-4de4-a399-5dae9b578f0c-kube-api-access-thqtc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:48.792758 master-1 kubenswrapper[4771]: I1011 10:35:48.792702 4771 reconciler_common.go:293] "Volume detached for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-proxy-ca-bundles\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:48.792758 master-1 kubenswrapper[4771]: I1011 10:35:48.792713 4771 reconciler_common.go:293] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-client-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:48.792758 master-1 kubenswrapper[4771]: I1011 10:35:48.792724 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62f55273-2711-4de4-a399-5dae9b578f0c-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:48.792758 master-1 kubenswrapper[4771]: I1011 10:35:48.792733 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62f55273-2711-4de4-a399-5dae9b578f0c-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:49.255553 master-1 kubenswrapper[4771]: I1011 10:35:49.254562 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-gwwz9"] Oct 11 10:35:49.255553 master-1 kubenswrapper[4771]: I1011 10:35:49.255071 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/community-operators-gwwz9" podUID="0b7d1d62-0062-47cd-a963-63893777198e" containerName="registry-server" containerID="cri-o://f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20" gracePeriod=2 Oct 11 10:35:49.308919 master-1 kubenswrapper[4771]: I1011 10:35:49.308699 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-controller-manager_controller-manager-5cf7cfc4c5-6jg5z_62f55273-2711-4de4-a399-5dae9b578f0c/controller-manager/0.log" Oct 11 10:35:49.308919 master-1 kubenswrapper[4771]: I1011 10:35:49.308820 4771 generic.go:334] "Generic (PLEG): container finished" podID="62f55273-2711-4de4-a399-5dae9b578f0c" containerID="3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3" exitCode=2 Oct 11 10:35:49.309301 master-1 kubenswrapper[4771]: I1011 10:35:49.308985 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" event={"ID":"62f55273-2711-4de4-a399-5dae9b578f0c","Type":"ContainerDied","Data":"3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3"} Oct 11 10:35:49.309301 master-1 kubenswrapper[4771]: I1011 10:35:49.309051 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" Oct 11 10:35:49.309301 master-1 kubenswrapper[4771]: I1011 10:35:49.309092 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z" event={"ID":"62f55273-2711-4de4-a399-5dae9b578f0c","Type":"ContainerDied","Data":"b1d9019c57f2da10f531e4404f9aaba506a3aa469ba7140f3ac23a4a053a6c4e"} Oct 11 10:35:49.309301 master-1 kubenswrapper[4771]: I1011 10:35:49.309139 4771 scope.go:117] "RemoveContainer" containerID="3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3" Oct 11 10:35:49.315991 master-1 kubenswrapper[4771]: I1011 10:35:49.315907 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:35:49.400481 master-1 kubenswrapper[4771]: I1011 10:35:49.400440 4771 scope.go:117] "RemoveContainer" containerID="3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3" Oct 11 10:35:49.401089 master-1 kubenswrapper[4771]: E1011 10:35:49.401036 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3\": container with ID starting with 3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3 not found: ID does not exist" containerID="3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3" Oct 11 10:35:49.401136 master-1 kubenswrapper[4771]: I1011 10:35:49.401098 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3"} err="failed to get container status \"3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3\": rpc error: code = NotFound desc = could not find container \"3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3\": container with ID starting with 3bc57acf5e1c9c9a4233c54fddb269b4320393069c0094b92821c7de170f91e3 not found: ID does not exist" Oct 11 10:35:49.461451 master-1 kubenswrapper[4771]: I1011 10:35:49.459496 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z"] Oct 11 10:35:49.466628 master-1 kubenswrapper[4771]: I1011 10:35:49.465912 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-controller-manager/controller-manager-5cf7cfc4c5-6jg5z"] Oct 11 10:35:49.500520 master-1 kubenswrapper[4771]: I1011 10:35:49.500454 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:49.500520 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:49.500520 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:49.500520 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:49.500520 master-1 kubenswrapper[4771]: I1011 10:35:49.500524 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:49.647800 master-1 kubenswrapper[4771]: I1011 10:35:49.647742 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/community-operators-t6wtm"] Oct 11 10:35:49.648549 master-1 kubenswrapper[4771]: E1011 10:35:49.648533 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="62f55273-2711-4de4-a399-5dae9b578f0c" containerName="controller-manager" Oct 11 10:35:49.648652 master-1 kubenswrapper[4771]: I1011 10:35:49.648641 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f55273-2711-4de4-a399-5dae9b578f0c" containerName="controller-manager" Oct 11 10:35:49.648807 master-1 kubenswrapper[4771]: I1011 10:35:49.648796 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="62f55273-2711-4de4-a399-5dae9b578f0c" containerName="controller-manager" Oct 11 10:35:49.650250 master-1 kubenswrapper[4771]: I1011 10:35:49.650232 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.654167 master-1 kubenswrapper[4771]: I1011 10:35:49.654113 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-marketplace"/"community-operators-dockercfg-sp4kx" Oct 11 10:35:49.660079 master-1 kubenswrapper[4771]: I1011 10:35:49.660029 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-t6wtm"] Oct 11 10:35:49.703543 master-1 kubenswrapper[4771]: I1011 10:35:49.703468 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/aebd88b2-f116-4ade-be8e-c293ccac533f-catalog-content\") pod \"community-operators-t6wtm\" (UID: \"aebd88b2-f116-4ade-be8e-c293ccac533f\") " pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.703869 master-1 kubenswrapper[4771]: I1011 10:35:49.703568 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/aebd88b2-f116-4ade-be8e-c293ccac533f-utilities\") pod \"community-operators-t6wtm\" (UID: \"aebd88b2-f116-4ade-be8e-c293ccac533f\") " pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.703869 master-1 kubenswrapper[4771]: I1011 10:35:49.703699 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbwc\" (UniqueName: \"kubernetes.io/projected/aebd88b2-f116-4ade-be8e-c293ccac533f-kube-api-access-7zbwc\") pod \"community-operators-t6wtm\" (UID: \"aebd88b2-f116-4ade-be8e-c293ccac533f\") " pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.722547 master-1 kubenswrapper[4771]: I1011 10:35:49.716261 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:35:49.804813 master-1 kubenswrapper[4771]: I1011 10:35:49.804741 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-catalog-content\") pod \"0b7d1d62-0062-47cd-a963-63893777198e\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " Oct 11 10:35:49.804813 master-1 kubenswrapper[4771]: I1011 10:35:49.804803 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6ptz\" (UniqueName: \"kubernetes.io/projected/0b7d1d62-0062-47cd-a963-63893777198e-kube-api-access-r6ptz\") pod \"0b7d1d62-0062-47cd-a963-63893777198e\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " Oct 11 10:35:49.805148 master-1 kubenswrapper[4771]: I1011 10:35:49.804903 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/aebd88b2-f116-4ade-be8e-c293ccac533f-utilities\") pod \"community-operators-t6wtm\" (UID: \"aebd88b2-f116-4ade-be8e-c293ccac533f\") " pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.805148 master-1 kubenswrapper[4771]: I1011 10:35:49.804992 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbwc\" (UniqueName: \"kubernetes.io/projected/aebd88b2-f116-4ade-be8e-c293ccac533f-kube-api-access-7zbwc\") pod \"community-operators-t6wtm\" (UID: \"aebd88b2-f116-4ade-be8e-c293ccac533f\") " pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.805148 master-1 kubenswrapper[4771]: I1011 10:35:49.805033 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/aebd88b2-f116-4ade-be8e-c293ccac533f-catalog-content\") pod \"community-operators-t6wtm\" (UID: \"aebd88b2-f116-4ade-be8e-c293ccac533f\") " pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.805678 master-1 kubenswrapper[4771]: I1011 10:35:49.805629 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/aebd88b2-f116-4ade-be8e-c293ccac533f-catalog-content\") pod \"community-operators-t6wtm\" (UID: \"aebd88b2-f116-4ade-be8e-c293ccac533f\") " pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.805862 master-1 kubenswrapper[4771]: I1011 10:35:49.805793 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/aebd88b2-f116-4ade-be8e-c293ccac533f-utilities\") pod \"community-operators-t6wtm\" (UID: \"aebd88b2-f116-4ade-be8e-c293ccac533f\") " pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.807760 master-1 kubenswrapper[4771]: I1011 10:35:49.807700 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7d1d62-0062-47cd-a963-63893777198e-kube-api-access-r6ptz" (OuterVolumeSpecName: "kube-api-access-r6ptz") pod "0b7d1d62-0062-47cd-a963-63893777198e" (UID: "0b7d1d62-0062-47cd-a963-63893777198e"). InnerVolumeSpecName "kube-api-access-r6ptz". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:35:49.832825 master-1 kubenswrapper[4771]: I1011 10:35:49.832732 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbwc\" (UniqueName: \"kubernetes.io/projected/aebd88b2-f116-4ade-be8e-c293ccac533f-kube-api-access-7zbwc\") pod \"community-operators-t6wtm\" (UID: \"aebd88b2-f116-4ade-be8e-c293ccac533f\") " pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:49.874718 master-1 kubenswrapper[4771]: I1011 10:35:49.874614 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "0b7d1d62-0062-47cd-a963-63893777198e" (UID: "0b7d1d62-0062-47cd-a963-63893777198e"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:35:49.905827 master-1 kubenswrapper[4771]: I1011 10:35:49.905780 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-utilities\") pod \"0b7d1d62-0062-47cd-a963-63893777198e\" (UID: \"0b7d1d62-0062-47cd-a963-63893777198e\") " Oct 11 10:35:49.906057 master-1 kubenswrapper[4771]: I1011 10:35:49.906014 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:49.906057 master-1 kubenswrapper[4771]: I1011 10:35:49.906041 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-r6ptz\" (UniqueName: \"kubernetes.io/projected/0b7d1d62-0062-47cd-a963-63893777198e-kube-api-access-r6ptz\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:49.907074 master-1 kubenswrapper[4771]: I1011 10:35:49.907013 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-utilities" (OuterVolumeSpecName: "utilities") pod "0b7d1d62-0062-47cd-a963-63893777198e" (UID: "0b7d1d62-0062-47cd-a963-63893777198e"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:35:49.944542 master-1 kubenswrapper[4771]: I1011 10:35:49.944462 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-controller-manager/controller-manager-77c7855cb4-qkp68"] Oct 11 10:35:49.944784 master-1 kubenswrapper[4771]: E1011 10:35:49.944674 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0b7d1d62-0062-47cd-a963-63893777198e" containerName="registry-server" Oct 11 10:35:49.944784 master-1 kubenswrapper[4771]: I1011 10:35:49.944689 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7d1d62-0062-47cd-a963-63893777198e" containerName="registry-server" Oct 11 10:35:49.944784 master-1 kubenswrapper[4771]: E1011 10:35:49.944701 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0b7d1d62-0062-47cd-a963-63893777198e" containerName="extract-utilities" Oct 11 10:35:49.944784 master-1 kubenswrapper[4771]: I1011 10:35:49.944709 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7d1d62-0062-47cd-a963-63893777198e" containerName="extract-utilities" Oct 11 10:35:49.944784 master-1 kubenswrapper[4771]: E1011 10:35:49.944723 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0b7d1d62-0062-47cd-a963-63893777198e" containerName="extract-content" Oct 11 10:35:49.944784 master-1 kubenswrapper[4771]: I1011 10:35:49.944733 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7d1d62-0062-47cd-a963-63893777198e" containerName="extract-content" Oct 11 10:35:49.945165 master-1 kubenswrapper[4771]: I1011 10:35:49.944832 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0b7d1d62-0062-47cd-a963-63893777198e" containerName="registry-server" Oct 11 10:35:49.945339 master-1 kubenswrapper[4771]: I1011 10:35:49.945285 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:49.948388 master-1 kubenswrapper[4771]: I1011 10:35:49.948302 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"client-ca" Oct 11 10:35:49.948592 master-1 kubenswrapper[4771]: I1011 10:35:49.948568 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-service-ca.crt" Oct 11 10:35:49.948700 master-1 kubenswrapper[4771]: I1011 10:35:49.948633 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"kube-root-ca.crt" Oct 11 10:35:49.948799 master-1 kubenswrapper[4771]: I1011 10:35:49.948701 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-controller-manager"/"serving-cert" Oct 11 10:35:49.949671 master-1 kubenswrapper[4771]: I1011 10:35:49.949620 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"config" Oct 11 10:35:49.955965 master-1 kubenswrapper[4771]: I1011 10:35:49.955903 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-global-ca" Oct 11 10:35:49.957861 master-1 kubenswrapper[4771]: I1011 10:35:49.957288 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-controller-manager/controller-manager-77c7855cb4-qkp68"] Oct 11 10:35:50.007769 master-1 kubenswrapper[4771]: I1011 10:35:50.007528 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlwv\" (UniqueName: \"kubernetes.io/projected/e23d9d43-9980-4c16-91c4-9fc0bca161e6-kube-api-access-fwlwv\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.007769 master-1 kubenswrapper[4771]: I1011 10:35:50.007647 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-client-ca\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.007769 master-1 kubenswrapper[4771]: I1011 10:35:50.007738 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-config\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.008256 master-1 kubenswrapper[4771]: I1011 10:35:50.007785 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e23d9d43-9980-4c16-91c4-9fc0bca161e6-serving-cert\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.008256 master-1 kubenswrapper[4771]: I1011 10:35:50.007896 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-proxy-ca-bundles\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.008256 master-1 kubenswrapper[4771]: I1011 10:35:50.007960 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/0b7d1d62-0062-47cd-a963-63893777198e-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:50.012943 master-1 kubenswrapper[4771]: I1011 10:35:50.012881 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:35:50.111014 master-1 kubenswrapper[4771]: I1011 10:35:50.108933 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-config\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.111014 master-1 kubenswrapper[4771]: I1011 10:35:50.108991 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e23d9d43-9980-4c16-91c4-9fc0bca161e6-serving-cert\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.111014 master-1 kubenswrapper[4771]: I1011 10:35:50.109053 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-proxy-ca-bundles\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.111014 master-1 kubenswrapper[4771]: I1011 10:35:50.109082 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlwv\" (UniqueName: \"kubernetes.io/projected/e23d9d43-9980-4c16-91c4-9fc0bca161e6-kube-api-access-fwlwv\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.111014 master-1 kubenswrapper[4771]: I1011 10:35:50.109106 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-client-ca\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.111014 master-1 kubenswrapper[4771]: I1011 10:35:50.110821 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-client-ca\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.111014 master-1 kubenswrapper[4771]: I1011 10:35:50.110946 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-config\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.112782 master-1 kubenswrapper[4771]: I1011 10:35:50.112736 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e23d9d43-9980-4c16-91c4-9fc0bca161e6-serving-cert\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.114438 master-1 kubenswrapper[4771]: I1011 10:35:50.114381 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-proxy-ca-bundles\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.134472 master-1 kubenswrapper[4771]: I1011 10:35:50.134170 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlwv\" (UniqueName: \"kubernetes.io/projected/e23d9d43-9980-4c16-91c4-9fc0bca161e6-kube-api-access-fwlwv\") pod \"controller-manager-77c7855cb4-qkp68\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.154933 master-1 kubenswrapper[4771]: I1011 10:35:50.154861 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/installer-6-master-1"] Oct 11 10:35:50.164912 master-1 kubenswrapper[4771]: I1011 10:35:50.163990 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.165643 master-1 kubenswrapper[4771]: I1011 10:35:50.165590 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-6-master-1"] Oct 11 10:35:50.166863 master-1 kubenswrapper[4771]: I1011 10:35:50.166840 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-kube-scheduler"/"installer-sa-dockercfg-js756" Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: I1011 10:35:50.246472 4771 patch_prober.go:28] interesting pod/metrics-server-65d86dff78-bg7lk container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: [+]metric-storage-ready ok Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: [+]metric-informer-sync ok Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: [+]metadata-informer-sync ok Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:50.246550 master-1 kubenswrapper[4771]: I1011 10:35:50.246537 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:50.272564 master-1 kubenswrapper[4771]: I1011 10:35:50.272437 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:50.311881 master-1 kubenswrapper[4771]: I1011 10:35:50.311788 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/67e39e90-67d5-40f4-ad76-1b32adf359ed-kube-api-access\") pod \"installer-6-master-1\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.312110 master-1 kubenswrapper[4771]: I1011 10:35:50.312048 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-var-lock\") pod \"installer-6-master-1\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.312208 master-1 kubenswrapper[4771]: I1011 10:35:50.312163 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-kubelet-dir\") pod \"installer-6-master-1\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.318574 master-1 kubenswrapper[4771]: I1011 10:35:50.318515 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b7d1d62-0062-47cd-a963-63893777198e" containerID="f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20" exitCode=0 Oct 11 10:35:50.318792 master-1 kubenswrapper[4771]: I1011 10:35:50.318632 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-gwwz9" event={"ID":"0b7d1d62-0062-47cd-a963-63893777198e","Type":"ContainerDied","Data":"f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20"} Oct 11 10:35:50.318897 master-1 kubenswrapper[4771]: I1011 10:35:50.318884 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-gwwz9" event={"ID":"0b7d1d62-0062-47cd-a963-63893777198e","Type":"ContainerDied","Data":"94df55f9d42e35f3eb12d9d840811113835d067c33b17a8f7670d61e212cd7f3"} Oct 11 10:35:50.319215 master-1 kubenswrapper[4771]: I1011 10:35:50.319016 4771 scope.go:117] "RemoveContainer" containerID="f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20" Oct 11 10:35:50.319365 master-1 kubenswrapper[4771]: I1011 10:35:50.319230 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-gwwz9" Oct 11 10:35:50.340413 master-1 kubenswrapper[4771]: I1011 10:35:50.340332 4771 scope.go:117] "RemoveContainer" containerID="d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80" Oct 11 10:35:50.362347 master-1 kubenswrapper[4771]: I1011 10:35:50.362273 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-gwwz9"] Oct 11 10:35:50.366682 master-1 kubenswrapper[4771]: I1011 10:35:50.366631 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/community-operators-gwwz9"] Oct 11 10:35:50.367378 master-1 kubenswrapper[4771]: I1011 10:35:50.367335 4771 scope.go:117] "RemoveContainer" containerID="87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865" Oct 11 10:35:50.438007 master-1 kubenswrapper[4771]: I1011 10:35:50.437947 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-kubelet-dir\") pod \"installer-6-master-1\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.438208 master-1 kubenswrapper[4771]: I1011 10:35:50.438022 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/67e39e90-67d5-40f4-ad76-1b32adf359ed-kube-api-access\") pod \"installer-6-master-1\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.438208 master-1 kubenswrapper[4771]: I1011 10:35:50.438105 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-var-lock\") pod \"installer-6-master-1\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.438713 master-1 kubenswrapper[4771]: I1011 10:35:50.438673 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-var-lock\") pod \"installer-6-master-1\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.438884 master-1 kubenswrapper[4771]: I1011 10:35:50.438859 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-kubelet-dir\") pod \"installer-6-master-1\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.445263 master-1 kubenswrapper[4771]: I1011 10:35:50.445170 4771 scope.go:117] "RemoveContainer" containerID="f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20" Oct 11 10:35:50.447742 master-1 kubenswrapper[4771]: E1011 10:35:50.447674 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20\": container with ID starting with f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20 not found: ID does not exist" containerID="f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20" Oct 11 10:35:50.447834 master-1 kubenswrapper[4771]: I1011 10:35:50.447748 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20"} err="failed to get container status \"f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20\": rpc error: code = NotFound desc = could not find container \"f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20\": container with ID starting with f22b32b9efc5550c030d0dab72ccfb608faf225ce28a60e3581e98388d7a1f20 not found: ID does not exist" Oct 11 10:35:50.447834 master-1 kubenswrapper[4771]: I1011 10:35:50.447789 4771 scope.go:117] "RemoveContainer" containerID="d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80" Oct 11 10:35:50.448678 master-1 kubenswrapper[4771]: E1011 10:35:50.448636 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80\": container with ID starting with d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80 not found: ID does not exist" containerID="d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80" Oct 11 10:35:50.448678 master-1 kubenswrapper[4771]: I1011 10:35:50.448666 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80"} err="failed to get container status \"d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80\": rpc error: code = NotFound desc = could not find container \"d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80\": container with ID starting with d9ec3c198a5bb93b32bff398f90b94c113b4f2ba904501149fb967a49c67ec80 not found: ID does not exist" Oct 11 10:35:50.448788 master-1 kubenswrapper[4771]: I1011 10:35:50.448684 4771 scope.go:117] "RemoveContainer" containerID="87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865" Oct 11 10:35:50.449099 master-1 kubenswrapper[4771]: E1011 10:35:50.449063 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865\": container with ID starting with 87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865 not found: ID does not exist" containerID="87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865" Oct 11 10:35:50.449099 master-1 kubenswrapper[4771]: I1011 10:35:50.449089 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865"} err="failed to get container status \"87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865\": rpc error: code = NotFound desc = could not find container \"87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865\": container with ID starting with 87c2f3c9c19accca6371d7e77d4bcd0cd514c07776fd9f98f32125516b8cf865 not found: ID does not exist" Oct 11 10:35:50.452675 master-1 kubenswrapper[4771]: I1011 10:35:50.452628 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7d1d62-0062-47cd-a963-63893777198e" path="/var/lib/kubelet/pods/0b7d1d62-0062-47cd-a963-63893777198e/volumes" Oct 11 10:35:50.454479 master-1 kubenswrapper[4771]: I1011 10:35:50.454342 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f55273-2711-4de4-a399-5dae9b578f0c" path="/var/lib/kubelet/pods/62f55273-2711-4de4-a399-5dae9b578f0c/volumes" Oct 11 10:35:50.462990 master-1 kubenswrapper[4771]: I1011 10:35:50.462938 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/67e39e90-67d5-40f4-ad76-1b32adf359ed-kube-api-access\") pod \"installer-6-master-1\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.474200 master-1 kubenswrapper[4771]: I1011 10:35:50.474137 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-t6wtm"] Oct 11 10:35:50.484912 master-1 kubenswrapper[4771]: W1011 10:35:50.484849 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebd88b2_f116_4ade_be8e_c293ccac533f.slice/crio-a0fc78bc0f59c467ca2c65c3c36d34a778b7360404a2f67daeae8fcd41742859 WatchSource:0}: Error finding container a0fc78bc0f59c467ca2c65c3c36d34a778b7360404a2f67daeae8fcd41742859: Status 404 returned error can't find the container with id a0fc78bc0f59c467ca2c65c3c36d34a778b7360404a2f67daeae8fcd41742859 Oct 11 10:35:50.495984 master-1 kubenswrapper[4771]: I1011 10:35:50.495944 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:35:50.497091 master-1 kubenswrapper[4771]: I1011 10:35:50.496896 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:50.497091 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:50.497091 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:50.497091 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:50.497091 master-1 kubenswrapper[4771]: I1011 10:35:50.496928 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:50.774102 master-1 kubenswrapper[4771]: I1011 10:35:50.774055 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-controller-manager/controller-manager-77c7855cb4-qkp68"] Oct 11 10:35:50.776320 master-1 kubenswrapper[4771]: W1011 10:35:50.776258 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23d9d43_9980_4c16_91c4_9fc0bca161e6.slice/crio-5577977e3fcec143fb9fe4819b109c252e41520252e8f2be4cdb67371fc4b2fd WatchSource:0}: Error finding container 5577977e3fcec143fb9fe4819b109c252e41520252e8f2be4cdb67371fc4b2fd: Status 404 returned error can't find the container with id 5577977e3fcec143fb9fe4819b109c252e41520252e8f2be4cdb67371fc4b2fd Oct 11 10:35:50.910708 master-1 kubenswrapper[4771]: I1011 10:35:50.910618 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/installer-6-master-1"] Oct 11 10:35:50.915516 master-1 kubenswrapper[4771]: W1011 10:35:50.915466 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod67e39e90_67d5_40f4_ad76_1b32adf359ed.slice/crio-c6be9acab2b46a0600e4b835c238bce39535b79bcb5079c1d439519d3a10a7ca WatchSource:0}: Error finding container c6be9acab2b46a0600e4b835c238bce39535b79bcb5079c1d439519d3a10a7ca: Status 404 returned error can't find the container with id c6be9acab2b46a0600e4b835c238bce39535b79bcb5079c1d439519d3a10a7ca Oct 11 10:35:51.332264 master-1 kubenswrapper[4771]: I1011 10:35:51.332089 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" event={"ID":"e23d9d43-9980-4c16-91c4-9fc0bca161e6","Type":"ContainerStarted","Data":"c570237a7e93abdb8d6cb4489a86eb34cb5e25db0de47a00c9bf05de3a2ba3c4"} Oct 11 10:35:51.332264 master-1 kubenswrapper[4771]: I1011 10:35:51.332158 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" event={"ID":"e23d9d43-9980-4c16-91c4-9fc0bca161e6","Type":"ContainerStarted","Data":"5577977e3fcec143fb9fe4819b109c252e41520252e8f2be4cdb67371fc4b2fd"} Oct 11 10:35:51.332869 master-1 kubenswrapper[4771]: I1011 10:35:51.332511 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:51.334996 master-1 kubenswrapper[4771]: I1011 10:35:51.334919 4771 generic.go:334] "Generic (PLEG): container finished" podID="aebd88b2-f116-4ade-be8e-c293ccac533f" containerID="ab992109a756b222e67d346b2be7bdaea651f492c702f39749fdc53e167dd28f" exitCode=0 Oct 11 10:35:51.335064 master-1 kubenswrapper[4771]: I1011 10:35:51.334991 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-t6wtm" event={"ID":"aebd88b2-f116-4ade-be8e-c293ccac533f","Type":"ContainerDied","Data":"ab992109a756b222e67d346b2be7bdaea651f492c702f39749fdc53e167dd28f"} Oct 11 10:35:51.335064 master-1 kubenswrapper[4771]: I1011 10:35:51.335058 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-t6wtm" event={"ID":"aebd88b2-f116-4ade-be8e-c293ccac533f","Type":"ContainerStarted","Data":"a0fc78bc0f59c467ca2c65c3c36d34a778b7360404a2f67daeae8fcd41742859"} Oct 11 10:35:51.338058 master-1 kubenswrapper[4771]: I1011 10:35:51.338022 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:35:51.339567 master-1 kubenswrapper[4771]: I1011 10:35:51.339530 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-6-master-1" event={"ID":"67e39e90-67d5-40f4-ad76-1b32adf359ed","Type":"ContainerStarted","Data":"4ac39222fba40ff7cbe78740b5c6cfd319b2ad66eef840556f4373378718527a"} Oct 11 10:35:51.339617 master-1 kubenswrapper[4771]: I1011 10:35:51.339568 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-6-master-1" event={"ID":"67e39e90-67d5-40f4-ad76-1b32adf359ed","Type":"ContainerStarted","Data":"c6be9acab2b46a0600e4b835c238bce39535b79bcb5079c1d439519d3a10a7ca"} Oct 11 10:35:51.352044 master-1 kubenswrapper[4771]: I1011 10:35:51.351808 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" podStartSLOduration=3.35177216 podStartE2EDuration="3.35177216s" podCreationTimestamp="2025-10-11 10:35:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:35:51.350974587 +0000 UTC m=+583.325201038" watchObservedRunningTime="2025-10-11 10:35:51.35177216 +0000 UTC m=+583.325998641" Oct 11 10:35:51.372288 master-1 kubenswrapper[4771]: I1011 10:35:51.372185 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/installer-6-master-1" podStartSLOduration=1.372149998 podStartE2EDuration="1.372149998s" podCreationTimestamp="2025-10-11 10:35:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:35:51.367805851 +0000 UTC m=+583.342032312" watchObservedRunningTime="2025-10-11 10:35:51.372149998 +0000 UTC m=+583.346376469" Oct 11 10:35:51.497720 master-1 kubenswrapper[4771]: I1011 10:35:51.497618 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:51.497720 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:51.497720 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:51.497720 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:51.497720 master-1 kubenswrapper[4771]: I1011 10:35:51.497723 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:51.848459 master-1 kubenswrapper[4771]: I1011 10:35:51.848168 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-xkrc6"] Oct 11 10:35:51.848727 master-1 kubenswrapper[4771]: I1011 10:35:51.848517 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-marketplace-xkrc6" podUID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerName="registry-server" containerID="cri-o://9f359af209588aa409904f71581bb63e20e019ac6f684b2bb1874bdc33d16458" gracePeriod=2 Oct 11 10:35:52.032549 master-1 kubenswrapper[4771]: I1011 10:35:52.032471 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-g8tm6"] Oct 11 10:35:52.032979 master-1 kubenswrapper[4771]: I1011 10:35:52.032918 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-operators-g8tm6" podUID="38131fcf-d407-4ba3-b7bf-471586bab887" containerName="registry-server" containerID="cri-o://c5ddefdc367347ae7e3aa6121d147be1b4ebca7be06e0180a8a6603ea9ef59cd" gracePeriod=2 Oct 11 10:35:52.243993 master-1 kubenswrapper[4771]: I1011 10:35:52.243910 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-marketplace-9ncpc"] Oct 11 10:35:52.244999 master-1 kubenswrapper[4771]: I1011 10:35:52.244944 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.248256 master-1 kubenswrapper[4771]: I1011 10:35:52.247928 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-marketplace"/"redhat-marketplace-dockercfg-mbhtz" Oct 11 10:35:52.261252 master-1 kubenswrapper[4771]: I1011 10:35:52.261186 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-9ncpc"] Oct 11 10:35:52.267453 master-1 kubenswrapper[4771]: I1011 10:35:52.265344 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/91e987bb-eae2-4f14-809d-1b1141882c7d-catalog-content\") pod \"redhat-marketplace-9ncpc\" (UID: \"91e987bb-eae2-4f14-809d-1b1141882c7d\") " pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.267453 master-1 kubenswrapper[4771]: I1011 10:35:52.265533 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/91e987bb-eae2-4f14-809d-1b1141882c7d-utilities\") pod \"redhat-marketplace-9ncpc\" (UID: \"91e987bb-eae2-4f14-809d-1b1141882c7d\") " pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.267453 master-1 kubenswrapper[4771]: I1011 10:35:52.265577 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gs9\" (UniqueName: \"kubernetes.io/projected/91e987bb-eae2-4f14-809d-1b1141882c7d-kube-api-access-z8gs9\") pod \"redhat-marketplace-9ncpc\" (UID: \"91e987bb-eae2-4f14-809d-1b1141882c7d\") " pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.349078 master-1 kubenswrapper[4771]: I1011 10:35:52.349013 4771 generic.go:334] "Generic (PLEG): container finished" podID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerID="9f359af209588aa409904f71581bb63e20e019ac6f684b2bb1874bdc33d16458" exitCode=0 Oct 11 10:35:52.349892 master-1 kubenswrapper[4771]: I1011 10:35:52.349096 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-xkrc6" event={"ID":"26005893-ecd8-4acb-8417-71a97ed97cbe","Type":"ContainerDied","Data":"9f359af209588aa409904f71581bb63e20e019ac6f684b2bb1874bdc33d16458"} Oct 11 10:35:52.352539 master-1 kubenswrapper[4771]: I1011 10:35:52.352419 4771 generic.go:334] "Generic (PLEG): container finished" podID="38131fcf-d407-4ba3-b7bf-471586bab887" containerID="c5ddefdc367347ae7e3aa6121d147be1b4ebca7be06e0180a8a6603ea9ef59cd" exitCode=0 Oct 11 10:35:52.352670 master-1 kubenswrapper[4771]: I1011 10:35:52.352519 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-g8tm6" event={"ID":"38131fcf-d407-4ba3-b7bf-471586bab887","Type":"ContainerDied","Data":"c5ddefdc367347ae7e3aa6121d147be1b4ebca7be06e0180a8a6603ea9ef59cd"} Oct 11 10:35:52.354852 master-1 kubenswrapper[4771]: I1011 10:35:52.354770 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-t6wtm" event={"ID":"aebd88b2-f116-4ade-be8e-c293ccac533f","Type":"ContainerStarted","Data":"909e21621c80ae096c10bbcb92430889443aedcf5f0b3f51e4c28a3ef5eaaddc"} Oct 11 10:35:52.366092 master-1 kubenswrapper[4771]: I1011 10:35:52.366045 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/91e987bb-eae2-4f14-809d-1b1141882c7d-catalog-content\") pod \"redhat-marketplace-9ncpc\" (UID: \"91e987bb-eae2-4f14-809d-1b1141882c7d\") " pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.367489 master-1 kubenswrapper[4771]: I1011 10:35:52.366238 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/91e987bb-eae2-4f14-809d-1b1141882c7d-utilities\") pod \"redhat-marketplace-9ncpc\" (UID: \"91e987bb-eae2-4f14-809d-1b1141882c7d\") " pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.367489 master-1 kubenswrapper[4771]: I1011 10:35:52.366271 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gs9\" (UniqueName: \"kubernetes.io/projected/91e987bb-eae2-4f14-809d-1b1141882c7d-kube-api-access-z8gs9\") pod \"redhat-marketplace-9ncpc\" (UID: \"91e987bb-eae2-4f14-809d-1b1141882c7d\") " pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.367489 master-1 kubenswrapper[4771]: I1011 10:35:52.367156 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/91e987bb-eae2-4f14-809d-1b1141882c7d-catalog-content\") pod \"redhat-marketplace-9ncpc\" (UID: \"91e987bb-eae2-4f14-809d-1b1141882c7d\") " pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.367489 master-1 kubenswrapper[4771]: I1011 10:35:52.367380 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/91e987bb-eae2-4f14-809d-1b1141882c7d-utilities\") pod \"redhat-marketplace-9ncpc\" (UID: \"91e987bb-eae2-4f14-809d-1b1141882c7d\") " pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.387263 master-1 kubenswrapper[4771]: I1011 10:35:52.387193 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gs9\" (UniqueName: \"kubernetes.io/projected/91e987bb-eae2-4f14-809d-1b1141882c7d-kube-api-access-z8gs9\") pod \"redhat-marketplace-9ncpc\" (UID: \"91e987bb-eae2-4f14-809d-1b1141882c7d\") " pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.406915 master-1 kubenswrapper[4771]: I1011 10:35:52.406788 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:35:52.426154 master-1 kubenswrapper[4771]: I1011 10:35:52.426083 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:35:52.461014 master-1 kubenswrapper[4771]: I1011 10:35:52.460949 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-operators-plxkp"] Oct 11 10:35:52.461310 master-1 kubenswrapper[4771]: E1011 10:35:52.461278 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="38131fcf-d407-4ba3-b7bf-471586bab887" containerName="extract-utilities" Oct 11 10:35:52.461310 master-1 kubenswrapper[4771]: I1011 10:35:52.461303 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="38131fcf-d407-4ba3-b7bf-471586bab887" containerName="extract-utilities" Oct 11 10:35:52.461412 master-1 kubenswrapper[4771]: E1011 10:35:52.461316 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerName="extract-utilities" Oct 11 10:35:52.461412 master-1 kubenswrapper[4771]: I1011 10:35:52.461327 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerName="extract-utilities" Oct 11 10:35:52.461412 master-1 kubenswrapper[4771]: E1011 10:35:52.461341 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerName="extract-content" Oct 11 10:35:52.461412 master-1 kubenswrapper[4771]: I1011 10:35:52.461378 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerName="extract-content" Oct 11 10:35:52.461412 master-1 kubenswrapper[4771]: E1011 10:35:52.461395 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerName="registry-server" Oct 11 10:35:52.461412 master-1 kubenswrapper[4771]: I1011 10:35:52.461404 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerName="registry-server" Oct 11 10:35:52.461576 master-1 kubenswrapper[4771]: E1011 10:35:52.461420 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="38131fcf-d407-4ba3-b7bf-471586bab887" containerName="extract-content" Oct 11 10:35:52.461576 master-1 kubenswrapper[4771]: I1011 10:35:52.461431 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="38131fcf-d407-4ba3-b7bf-471586bab887" containerName="extract-content" Oct 11 10:35:52.461576 master-1 kubenswrapper[4771]: E1011 10:35:52.461450 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="38131fcf-d407-4ba3-b7bf-471586bab887" containerName="registry-server" Oct 11 10:35:52.461576 master-1 kubenswrapper[4771]: I1011 10:35:52.461459 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="38131fcf-d407-4ba3-b7bf-471586bab887" containerName="registry-server" Oct 11 10:35:52.461679 master-1 kubenswrapper[4771]: I1011 10:35:52.461614 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="26005893-ecd8-4acb-8417-71a97ed97cbe" containerName="registry-server" Oct 11 10:35:52.461679 master-1 kubenswrapper[4771]: I1011 10:35:52.461638 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="38131fcf-d407-4ba3-b7bf-471586bab887" containerName="registry-server" Oct 11 10:35:52.462625 master-1 kubenswrapper[4771]: I1011 10:35:52.462590 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-plxkp"] Oct 11 10:35:52.462789 master-1 kubenswrapper[4771]: I1011 10:35:52.462735 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.465655 master-1 kubenswrapper[4771]: I1011 10:35:52.465472 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-marketplace"/"redhat-operators-dockercfg-bpf7n" Oct 11 10:35:52.467651 master-1 kubenswrapper[4771]: I1011 10:35:52.467603 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/46805d49-0205-4427-9403-2fd481f36555-catalog-content\") pod \"redhat-operators-plxkp\" (UID: \"46805d49-0205-4427-9403-2fd481f36555\") " pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.467739 master-1 kubenswrapper[4771]: I1011 10:35:52.467681 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/46805d49-0205-4427-9403-2fd481f36555-utilities\") pod \"redhat-operators-plxkp\" (UID: \"46805d49-0205-4427-9403-2fd481f36555\") " pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.495931 master-1 kubenswrapper[4771]: I1011 10:35:52.495850 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:52.495931 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:52.495931 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:52.495931 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:52.495931 master-1 kubenswrapper[4771]: I1011 10:35:52.495922 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:52.568990 master-1 kubenswrapper[4771]: I1011 10:35:52.568907 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-utilities\") pod \"38131fcf-d407-4ba3-b7bf-471586bab887\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " Oct 11 10:35:52.569398 master-1 kubenswrapper[4771]: I1011 10:35:52.569253 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-catalog-content\") pod \"26005893-ecd8-4acb-8417-71a97ed97cbe\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " Oct 11 10:35:52.569398 master-1 kubenswrapper[4771]: I1011 10:35:52.569290 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpmjh\" (UniqueName: \"kubernetes.io/projected/38131fcf-d407-4ba3-b7bf-471586bab887-kube-api-access-gpmjh\") pod \"38131fcf-d407-4ba3-b7bf-471586bab887\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " Oct 11 10:35:52.569398 master-1 kubenswrapper[4771]: I1011 10:35:52.569313 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqnhk\" (UniqueName: \"kubernetes.io/projected/26005893-ecd8-4acb-8417-71a97ed97cbe-kube-api-access-hqnhk\") pod \"26005893-ecd8-4acb-8417-71a97ed97cbe\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " Oct 11 10:35:52.569398 master-1 kubenswrapper[4771]: I1011 10:35:52.569333 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-catalog-content\") pod \"38131fcf-d407-4ba3-b7bf-471586bab887\" (UID: \"38131fcf-d407-4ba3-b7bf-471586bab887\") " Oct 11 10:35:52.569398 master-1 kubenswrapper[4771]: I1011 10:35:52.569376 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-utilities\") pod \"26005893-ecd8-4acb-8417-71a97ed97cbe\" (UID: \"26005893-ecd8-4acb-8417-71a97ed97cbe\") " Oct 11 10:35:52.569562 master-1 kubenswrapper[4771]: I1011 10:35:52.569461 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/46805d49-0205-4427-9403-2fd481f36555-utilities\") pod \"redhat-operators-plxkp\" (UID: \"46805d49-0205-4427-9403-2fd481f36555\") " pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.569562 master-1 kubenswrapper[4771]: I1011 10:35:52.569497 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjxq\" (UniqueName: \"kubernetes.io/projected/46805d49-0205-4427-9403-2fd481f36555-kube-api-access-4zjxq\") pod \"redhat-operators-plxkp\" (UID: \"46805d49-0205-4427-9403-2fd481f36555\") " pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.569562 master-1 kubenswrapper[4771]: I1011 10:35:52.569551 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/46805d49-0205-4427-9403-2fd481f36555-catalog-content\") pod \"redhat-operators-plxkp\" (UID: \"46805d49-0205-4427-9403-2fd481f36555\") " pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.570065 master-1 kubenswrapper[4771]: I1011 10:35:52.570024 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/46805d49-0205-4427-9403-2fd481f36555-catalog-content\") pod \"redhat-operators-plxkp\" (UID: \"46805d49-0205-4427-9403-2fd481f36555\") " pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.571077 master-1 kubenswrapper[4771]: I1011 10:35:52.571023 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/46805d49-0205-4427-9403-2fd481f36555-utilities\") pod \"redhat-operators-plxkp\" (UID: \"46805d49-0205-4427-9403-2fd481f36555\") " pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.571139 master-1 kubenswrapper[4771]: I1011 10:35:52.571114 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-utilities" (OuterVolumeSpecName: "utilities") pod "38131fcf-d407-4ba3-b7bf-471586bab887" (UID: "38131fcf-d407-4ba3-b7bf-471586bab887"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:35:52.572289 master-1 kubenswrapper[4771]: I1011 10:35:52.572199 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-utilities" (OuterVolumeSpecName: "utilities") pod "26005893-ecd8-4acb-8417-71a97ed97cbe" (UID: "26005893-ecd8-4acb-8417-71a97ed97cbe"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:35:52.573810 master-1 kubenswrapper[4771]: I1011 10:35:52.573760 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26005893-ecd8-4acb-8417-71a97ed97cbe-kube-api-access-hqnhk" (OuterVolumeSpecName: "kube-api-access-hqnhk") pod "26005893-ecd8-4acb-8417-71a97ed97cbe" (UID: "26005893-ecd8-4acb-8417-71a97ed97cbe"). InnerVolumeSpecName "kube-api-access-hqnhk". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:35:52.574023 master-1 kubenswrapper[4771]: I1011 10:35:52.573980 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38131fcf-d407-4ba3-b7bf-471586bab887-kube-api-access-gpmjh" (OuterVolumeSpecName: "kube-api-access-gpmjh") pod "38131fcf-d407-4ba3-b7bf-471586bab887" (UID: "38131fcf-d407-4ba3-b7bf-471586bab887"). InnerVolumeSpecName "kube-api-access-gpmjh". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:35:52.592363 master-1 kubenswrapper[4771]: I1011 10:35:52.592285 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:35:52.594045 master-1 kubenswrapper[4771]: I1011 10:35:52.593976 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "26005893-ecd8-4acb-8417-71a97ed97cbe" (UID: "26005893-ecd8-4acb-8417-71a97ed97cbe"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:35:52.670751 master-1 kubenswrapper[4771]: I1011 10:35:52.670536 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjxq\" (UniqueName: \"kubernetes.io/projected/46805d49-0205-4427-9403-2fd481f36555-kube-api-access-4zjxq\") pod \"redhat-operators-plxkp\" (UID: \"46805d49-0205-4427-9403-2fd481f36555\") " pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.670751 master-1 kubenswrapper[4771]: I1011 10:35:52.670622 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:52.670751 master-1 kubenswrapper[4771]: I1011 10:35:52.670641 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-gpmjh\" (UniqueName: \"kubernetes.io/projected/38131fcf-d407-4ba3-b7bf-471586bab887-kube-api-access-gpmjh\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:52.670751 master-1 kubenswrapper[4771]: I1011 10:35:52.670655 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-hqnhk\" (UniqueName: \"kubernetes.io/projected/26005893-ecd8-4acb-8417-71a97ed97cbe-kube-api-access-hqnhk\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:52.670751 master-1 kubenswrapper[4771]: I1011 10:35:52.670669 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/26005893-ecd8-4acb-8417-71a97ed97cbe-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:52.670751 master-1 kubenswrapper[4771]: I1011 10:35:52.670680 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:52.693696 master-1 kubenswrapper[4771]: I1011 10:35:52.693599 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "38131fcf-d407-4ba3-b7bf-471586bab887" (UID: "38131fcf-d407-4ba3-b7bf-471586bab887"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:35:52.694512 master-1 kubenswrapper[4771]: I1011 10:35:52.694471 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjxq\" (UniqueName: \"kubernetes.io/projected/46805d49-0205-4427-9403-2fd481f36555-kube-api-access-4zjxq\") pod \"redhat-operators-plxkp\" (UID: \"46805d49-0205-4427-9403-2fd481f36555\") " pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:52.771609 master-1 kubenswrapper[4771]: I1011 10:35:52.771525 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/38131fcf-d407-4ba3-b7bf-471586bab887-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 10:35:52.783611 master-1 kubenswrapper[4771]: I1011 10:35:52.783504 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:35:53.033677 master-1 kubenswrapper[4771]: I1011 10:35:53.033578 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-9ncpc"] Oct 11 10:35:53.040432 master-1 kubenswrapper[4771]: W1011 10:35:53.040317 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e987bb_eae2_4f14_809d_1b1141882c7d.slice/crio-b754598af606a210c7b5b92cfa803febec63a5f9e91702fa35cc1b301b1a3cb0 WatchSource:0}: Error finding container b754598af606a210c7b5b92cfa803febec63a5f9e91702fa35cc1b301b1a3cb0: Status 404 returned error can't find the container with id b754598af606a210c7b5b92cfa803febec63a5f9e91702fa35cc1b301b1a3cb0 Oct 11 10:35:53.241021 master-1 kubenswrapper[4771]: I1011 10:35:53.240842 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-plxkp"] Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: I1011 10:35:53.244003 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:53.244048 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:53.246421 master-1 kubenswrapper[4771]: I1011 10:35:53.244054 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:53.253185 master-1 kubenswrapper[4771]: W1011 10:35:53.253105 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46805d49_0205_4427_9403_2fd481f36555.slice/crio-5c9680c9e6353071262fafb6e5336eeb9cd257540fc8b17b829f5b62da03bd5b WatchSource:0}: Error finding container 5c9680c9e6353071262fafb6e5336eeb9cd257540fc8b17b829f5b62da03bd5b: Status 404 returned error can't find the container with id 5c9680c9e6353071262fafb6e5336eeb9cd257540fc8b17b829f5b62da03bd5b Oct 11 10:35:53.366203 master-1 kubenswrapper[4771]: I1011 10:35:53.366166 4771 generic.go:334] "Generic (PLEG): container finished" podID="91e987bb-eae2-4f14-809d-1b1141882c7d" containerID="e95237df12bc6ebde5d45bedbd51537f6d7df95a3d30b122ee215229afe6e48c" exitCode=0 Oct 11 10:35:53.366975 master-1 kubenswrapper[4771]: I1011 10:35:53.366236 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-9ncpc" event={"ID":"91e987bb-eae2-4f14-809d-1b1141882c7d","Type":"ContainerDied","Data":"e95237df12bc6ebde5d45bedbd51537f6d7df95a3d30b122ee215229afe6e48c"} Oct 11 10:35:53.368170 master-1 kubenswrapper[4771]: I1011 10:35:53.367476 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-9ncpc" event={"ID":"91e987bb-eae2-4f14-809d-1b1141882c7d","Type":"ContainerStarted","Data":"b754598af606a210c7b5b92cfa803febec63a5f9e91702fa35cc1b301b1a3cb0"} Oct 11 10:35:53.369421 master-1 kubenswrapper[4771]: I1011 10:35:53.369333 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-plxkp" event={"ID":"46805d49-0205-4427-9403-2fd481f36555","Type":"ContainerStarted","Data":"5c9680c9e6353071262fafb6e5336eeb9cd257540fc8b17b829f5b62da03bd5b"} Oct 11 10:35:53.373184 master-1 kubenswrapper[4771]: I1011 10:35:53.373127 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-xkrc6" event={"ID":"26005893-ecd8-4acb-8417-71a97ed97cbe","Type":"ContainerDied","Data":"42678277150d23882615afd583505d1ee80fbc936870ab20c76affe3a676bd4c"} Oct 11 10:35:53.373184 master-1 kubenswrapper[4771]: I1011 10:35:53.373182 4771 scope.go:117] "RemoveContainer" containerID="9f359af209588aa409904f71581bb63e20e019ac6f684b2bb1874bdc33d16458" Oct 11 10:35:53.373403 master-1 kubenswrapper[4771]: I1011 10:35:53.373210 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-xkrc6" Oct 11 10:35:53.377283 master-1 kubenswrapper[4771]: I1011 10:35:53.377227 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-g8tm6" event={"ID":"38131fcf-d407-4ba3-b7bf-471586bab887","Type":"ContainerDied","Data":"fd83c4d331d341ca058f07884e0c753dec2509d54999da528657ce66ee47354c"} Oct 11 10:35:53.377438 master-1 kubenswrapper[4771]: I1011 10:35:53.377247 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-g8tm6" Oct 11 10:35:53.379976 master-1 kubenswrapper[4771]: I1011 10:35:53.379941 4771 generic.go:334] "Generic (PLEG): container finished" podID="aebd88b2-f116-4ade-be8e-c293ccac533f" containerID="909e21621c80ae096c10bbcb92430889443aedcf5f0b3f51e4c28a3ef5eaaddc" exitCode=0 Oct 11 10:35:53.380088 master-1 kubenswrapper[4771]: I1011 10:35:53.380021 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-t6wtm" event={"ID":"aebd88b2-f116-4ade-be8e-c293ccac533f","Type":"ContainerDied","Data":"909e21621c80ae096c10bbcb92430889443aedcf5f0b3f51e4c28a3ef5eaaddc"} Oct 11 10:35:53.409132 master-1 kubenswrapper[4771]: I1011 10:35:53.409084 4771 scope.go:117] "RemoveContainer" containerID="4bde2f0bff6002ac88c69a20de25c24e27ed2402f74ddf6b6f429bda18e25de4" Oct 11 10:35:53.458404 master-1 kubenswrapper[4771]: I1011 10:35:53.458179 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-xkrc6"] Oct 11 10:35:53.461587 master-1 kubenswrapper[4771]: I1011 10:35:53.461547 4771 scope.go:117] "RemoveContainer" containerID="142add763393fde94b8ed6a34c3ef572a32e34909b409ad71cf3570c801fa30d" Oct 11 10:35:53.463882 master-1 kubenswrapper[4771]: I1011 10:35:53.463834 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-marketplace-xkrc6"] Oct 11 10:35:53.477578 master-1 kubenswrapper[4771]: I1011 10:35:53.477511 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-g8tm6"] Oct 11 10:35:53.482751 master-1 kubenswrapper[4771]: I1011 10:35:53.482696 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-operators-g8tm6"] Oct 11 10:35:53.497949 master-1 kubenswrapper[4771]: I1011 10:35:53.497832 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:53.497949 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:53.497949 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:53.497949 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:53.498130 master-1 kubenswrapper[4771]: I1011 10:35:53.497935 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:53.517068 master-1 kubenswrapper[4771]: I1011 10:35:53.517029 4771 scope.go:117] "RemoveContainer" containerID="c5ddefdc367347ae7e3aa6121d147be1b4ebca7be06e0180a8a6603ea9ef59cd" Oct 11 10:35:53.536863 master-1 kubenswrapper[4771]: I1011 10:35:53.536793 4771 scope.go:117] "RemoveContainer" containerID="10eecae7180584a993b9109e41de9729732ec8af959166bad8fe7ba33a08f83b" Oct 11 10:35:53.555125 master-1 kubenswrapper[4771]: I1011 10:35:53.555019 4771 scope.go:117] "RemoveContainer" containerID="46478dfa370c61d5e583543ca4a34b66afd1e95ecf434515eb16283cfe8a52de" Oct 11 10:35:54.396799 master-1 kubenswrapper[4771]: I1011 10:35:54.395882 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-t6wtm" event={"ID":"aebd88b2-f116-4ade-be8e-c293ccac533f","Type":"ContainerStarted","Data":"99d4f42a37c07831bbeb02c6d60d0ab3cac5eecfde12522617d4ee8db3495770"} Oct 11 10:35:54.399884 master-1 kubenswrapper[4771]: I1011 10:35:54.399032 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-9ncpc" event={"ID":"91e987bb-eae2-4f14-809d-1b1141882c7d","Type":"ContainerStarted","Data":"31c4e7524b25a68b245a3075aa05d28b3a09ace752abb0471cf1fe03cea33242"} Oct 11 10:35:54.402177 master-1 kubenswrapper[4771]: I1011 10:35:54.402050 4771 generic.go:334] "Generic (PLEG): container finished" podID="46805d49-0205-4427-9403-2fd481f36555" containerID="f3eab5f2de54810f88dae9b35d2a1d8e381f1f4f5815ea9b946ed648d69d5f2a" exitCode=0 Oct 11 10:35:54.402331 master-1 kubenswrapper[4771]: I1011 10:35:54.402165 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-plxkp" event={"ID":"46805d49-0205-4427-9403-2fd481f36555","Type":"ContainerDied","Data":"f3eab5f2de54810f88dae9b35d2a1d8e381f1f4f5815ea9b946ed648d69d5f2a"} Oct 11 10:35:54.449249 master-1 kubenswrapper[4771]: I1011 10:35:54.449002 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26005893-ecd8-4acb-8417-71a97ed97cbe" path="/var/lib/kubelet/pods/26005893-ecd8-4acb-8417-71a97ed97cbe/volumes" Oct 11 10:35:54.450629 master-1 kubenswrapper[4771]: I1011 10:35:54.450559 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38131fcf-d407-4ba3-b7bf-471586bab887" path="/var/lib/kubelet/pods/38131fcf-d407-4ba3-b7bf-471586bab887/volumes" Oct 11 10:35:54.456533 master-1 kubenswrapper[4771]: I1011 10:35:54.456432 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/community-operators-t6wtm" podStartSLOduration=3.047163613 podStartE2EDuration="5.456400015s" podCreationTimestamp="2025-10-11 10:35:49 +0000 UTC" firstStartedPulling="2025-10-11 10:35:51.33678346 +0000 UTC m=+583.311009901" lastFinishedPulling="2025-10-11 10:35:53.746019822 +0000 UTC m=+585.720246303" observedRunningTime="2025-10-11 10:35:54.425838833 +0000 UTC m=+586.400065334" watchObservedRunningTime="2025-10-11 10:35:54.456400015 +0000 UTC m=+586.430626496" Oct 11 10:35:54.496605 master-1 kubenswrapper[4771]: I1011 10:35:54.496517 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:54.496605 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:54.496605 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:54.496605 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:54.497156 master-1 kubenswrapper[4771]: I1011 10:35:54.496611 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:55.417006 master-1 kubenswrapper[4771]: I1011 10:35:55.416907 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-plxkp" event={"ID":"46805d49-0205-4427-9403-2fd481f36555","Type":"ContainerStarted","Data":"31e117d1e88c620e780185b41834b342b6ae4b1958b78cceb77d28cf75bab545"} Oct 11 10:35:55.421660 master-1 kubenswrapper[4771]: I1011 10:35:55.421581 4771 generic.go:334] "Generic (PLEG): container finished" podID="91e987bb-eae2-4f14-809d-1b1141882c7d" containerID="31c4e7524b25a68b245a3075aa05d28b3a09ace752abb0471cf1fe03cea33242" exitCode=0 Oct 11 10:35:55.421823 master-1 kubenswrapper[4771]: I1011 10:35:55.421660 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-9ncpc" event={"ID":"91e987bb-eae2-4f14-809d-1b1141882c7d","Type":"ContainerDied","Data":"31c4e7524b25a68b245a3075aa05d28b3a09ace752abb0471cf1fe03cea33242"} Oct 11 10:35:55.497102 master-1 kubenswrapper[4771]: I1011 10:35:55.496910 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:55.497102 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:55.497102 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:55.497102 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:55.497102 master-1 kubenswrapper[4771]: I1011 10:35:55.497008 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:56.431146 master-1 kubenswrapper[4771]: I1011 10:35:56.431060 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-9ncpc" event={"ID":"91e987bb-eae2-4f14-809d-1b1141882c7d","Type":"ContainerStarted","Data":"6080741b4d7eae18a44d1a08030b7a435ed3cfe3526dea8c026a6b1cfdfdc86d"} Oct 11 10:35:56.433664 master-1 kubenswrapper[4771]: I1011 10:35:56.433589 4771 generic.go:334] "Generic (PLEG): container finished" podID="46805d49-0205-4427-9403-2fd481f36555" containerID="31e117d1e88c620e780185b41834b342b6ae4b1958b78cceb77d28cf75bab545" exitCode=0 Oct 11 10:35:56.433664 master-1 kubenswrapper[4771]: I1011 10:35:56.433660 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-plxkp" event={"ID":"46805d49-0205-4427-9403-2fd481f36555","Type":"ContainerDied","Data":"31e117d1e88c620e780185b41834b342b6ae4b1958b78cceb77d28cf75bab545"} Oct 11 10:35:56.496998 master-1 kubenswrapper[4771]: I1011 10:35:56.496898 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:56.496998 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:56.496998 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:56.496998 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:56.497481 master-1 kubenswrapper[4771]: I1011 10:35:56.497004 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:57.443278 master-1 kubenswrapper[4771]: I1011 10:35:57.443178 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-plxkp" event={"ID":"46805d49-0205-4427-9403-2fd481f36555","Type":"ContainerStarted","Data":"8b7a5f47530c9f5fd8e4c6cf8d74b538da46568b8286fc7969d4b9f71e24aa8c"} Oct 11 10:35:57.496646 master-1 kubenswrapper[4771]: I1011 10:35:57.496552 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:57.496646 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:57.496646 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:57.496646 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:57.496646 master-1 kubenswrapper[4771]: I1011 10:35:57.496639 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: I1011 10:35:58.244586 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:35:58.244652 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:35:58.245792 master-1 kubenswrapper[4771]: I1011 10:35:58.244671 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:58.497611 master-1 kubenswrapper[4771]: I1011 10:35:58.497467 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:58.497611 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:58.497611 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:58.497611 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:58.497611 master-1 kubenswrapper[4771]: I1011 10:35:58.497558 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:35:59.498639 master-1 kubenswrapper[4771]: I1011 10:35:59.498543 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:35:59.498639 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:35:59.498639 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:35:59.498639 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:35:59.499636 master-1 kubenswrapper[4771]: I1011 10:35:59.498638 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:00.013586 master-1 kubenswrapper[4771]: I1011 10:36:00.013455 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:36:00.013586 master-1 kubenswrapper[4771]: I1011 10:36:00.013561 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:36:00.062295 master-1 kubenswrapper[4771]: I1011 10:36:00.062221 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:36:00.497498 master-1 kubenswrapper[4771]: I1011 10:36:00.497409 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:00.497498 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:00.497498 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:00.497498 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:00.497792 master-1 kubenswrapper[4771]: I1011 10:36:00.497506 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:00.523163 master-1 kubenswrapper[4771]: I1011 10:36:00.523097 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/community-operators-t6wtm" Oct 11 10:36:01.496866 master-1 kubenswrapper[4771]: I1011 10:36:01.496794 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:01.496866 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:01.496866 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:01.496866 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:01.497267 master-1 kubenswrapper[4771]: I1011 10:36:01.496868 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:02.497617 master-1 kubenswrapper[4771]: I1011 10:36:02.497498 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:02.497617 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:02.497617 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:02.497617 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:02.497617 master-1 kubenswrapper[4771]: I1011 10:36:02.497606 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:02.593609 master-1 kubenswrapper[4771]: I1011 10:36:02.593525 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:36:02.593609 master-1 kubenswrapper[4771]: I1011 10:36:02.593603 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:36:02.669769 master-1 kubenswrapper[4771]: I1011 10:36:02.669710 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:36:02.784688 master-1 kubenswrapper[4771]: I1011 10:36:02.784490 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:36:02.784688 master-1 kubenswrapper[4771]: I1011 10:36:02.784577 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: I1011 10:36:03.243252 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:36:03.243380 master-1 kubenswrapper[4771]: I1011 10:36:03.243373 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:03.497482 master-1 kubenswrapper[4771]: I1011 10:36:03.497224 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:03.497482 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:03.497482 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:03.497482 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:03.497482 master-1 kubenswrapper[4771]: I1011 10:36:03.497321 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:03.551611 master-1 kubenswrapper[4771]: I1011 10:36:03.551519 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-marketplace-9ncpc" Oct 11 10:36:03.834316 master-1 kubenswrapper[4771]: I1011 10:36:03.834235 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-marketplace/redhat-operators-plxkp" podUID="46805d49-0205-4427-9403-2fd481f36555" containerName="registry-server" probeResult="failure" output=< Oct 11 10:36:03.834316 master-1 kubenswrapper[4771]: timeout: failed to connect service ":50051" within 1s Oct 11 10:36:03.834316 master-1 kubenswrapper[4771]: > Oct 11 10:36:04.496893 master-1 kubenswrapper[4771]: I1011 10:36:04.496799 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:04.496893 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:04.496893 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:04.496893 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:04.497455 master-1 kubenswrapper[4771]: I1011 10:36:04.496904 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:05.496859 master-1 kubenswrapper[4771]: I1011 10:36:05.496779 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:05.496859 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:05.496859 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:05.496859 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:05.497413 master-1 kubenswrapper[4771]: I1011 10:36:05.496864 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:06.497148 master-1 kubenswrapper[4771]: I1011 10:36:06.497085 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:06.497148 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:06.497148 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:06.497148 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:06.498097 master-1 kubenswrapper[4771]: I1011 10:36:06.498059 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:07.497207 master-1 kubenswrapper[4771]: I1011 10:36:07.497150 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:07.497207 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:07.497207 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:07.497207 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:07.498171 master-1 kubenswrapper[4771]: I1011 10:36:07.497524 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: I1011 10:36:08.245503 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:36:08.245592 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:36:08.247992 master-1 kubenswrapper[4771]: I1011 10:36:08.245601 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:08.498466 master-1 kubenswrapper[4771]: I1011 10:36:08.498218 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:08.498466 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:08.498466 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:08.498466 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:08.498466 master-1 kubenswrapper[4771]: I1011 10:36:08.498318 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:09.497987 master-1 kubenswrapper[4771]: I1011 10:36:09.497789 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:09.497987 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:09.497987 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:09.497987 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:09.497987 master-1 kubenswrapper[4771]: I1011 10:36:09.497910 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:10.247676 master-1 kubenswrapper[4771]: I1011 10:36:10.247474 4771 patch_prober.go:28] interesting pod/metrics-server-65d86dff78-bg7lk container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:36:10.247676 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:36:10.247676 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:36:10.247676 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:36:10.247676 master-1 kubenswrapper[4771]: [+]metric-storage-ready ok Oct 11 10:36:10.247676 master-1 kubenswrapper[4771]: [+]metric-informer-sync ok Oct 11 10:36:10.247676 master-1 kubenswrapper[4771]: [+]metadata-informer-sync ok Oct 11 10:36:10.247676 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:36:10.247676 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:36:10.248470 master-1 kubenswrapper[4771]: I1011 10:36:10.247684 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:10.497259 master-1 kubenswrapper[4771]: I1011 10:36:10.497177 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:10.497259 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:10.497259 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:10.497259 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:10.497800 master-1 kubenswrapper[4771]: I1011 10:36:10.497271 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:11.497791 master-1 kubenswrapper[4771]: I1011 10:36:11.497674 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:11.497791 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:11.497791 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:11.497791 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:11.498557 master-1 kubenswrapper[4771]: I1011 10:36:11.497794 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:12.497812 master-1 kubenswrapper[4771]: I1011 10:36:12.497681 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:12.497812 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:12.497812 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:12.497812 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:12.497812 master-1 kubenswrapper[4771]: I1011 10:36:12.497804 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:12.540862 master-1 kubenswrapper[4771]: I1011 10:36:12.540509 4771 scope.go:117] "RemoveContainer" containerID="4f12c3536caf37d890a386fecb2c94e5fc57775602e9a539771326b213c3ae7e" Oct 11 10:36:12.561829 master-1 kubenswrapper[4771]: I1011 10:36:12.561760 4771 scope.go:117] "RemoveContainer" containerID="0400db595d18039edaf6ab7ccb3c1b1a3510ae9588fc33a6a91a15e993a6d1a4" Oct 11 10:36:12.586057 master-1 kubenswrapper[4771]: I1011 10:36:12.585989 4771 scope.go:117] "RemoveContainer" containerID="27a52449e5ec1bd52177b8ae4e5229c8bc4e5a7be149b07a0e7cb307be3932da" Oct 11 10:36:12.612621 master-1 kubenswrapper[4771]: I1011 10:36:12.612550 4771 scope.go:117] "RemoveContainer" containerID="2a73de07f276bd8a0b93475494fdae31f01c7c950b265a424f35d3d72462410c" Oct 11 10:36:12.846130 master-1 kubenswrapper[4771]: I1011 10:36:12.846054 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:36:12.910259 master-1 kubenswrapper[4771]: I1011 10:36:12.910158 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-operators-plxkp" Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: I1011 10:36:13.242517 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:36:13.242644 master-1 kubenswrapper[4771]: I1011 10:36:13.242599 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:13.497135 master-1 kubenswrapper[4771]: I1011 10:36:13.496981 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:13.497135 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:13.497135 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:13.497135 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:13.497135 master-1 kubenswrapper[4771]: I1011 10:36:13.497092 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:14.498150 master-1 kubenswrapper[4771]: I1011 10:36:14.498040 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:14.498150 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:14.498150 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:14.498150 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:14.498150 master-1 kubenswrapper[4771]: I1011 10:36:14.498136 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:15.497013 master-1 kubenswrapper[4771]: I1011 10:36:15.496910 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:15.497013 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:15.497013 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:15.497013 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:15.497539 master-1 kubenswrapper[4771]: I1011 10:36:15.497023 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:16.498148 master-1 kubenswrapper[4771]: I1011 10:36:16.498036 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:16.498148 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:16.498148 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:16.498148 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:16.499352 master-1 kubenswrapper[4771]: I1011 10:36:16.498170 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:17.497141 master-1 kubenswrapper[4771]: I1011 10:36:17.497051 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:17.497141 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:17.497141 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:17.497141 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:17.497663 master-1 kubenswrapper[4771]: I1011 10:36:17.497164 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: I1011 10:36:18.241751 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:36:18.241830 master-1 kubenswrapper[4771]: I1011 10:36:18.241814 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:18.498517 master-1 kubenswrapper[4771]: I1011 10:36:18.498265 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:18.498517 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:18.498517 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:18.498517 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:18.498517 master-1 kubenswrapper[4771]: I1011 10:36:18.498417 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:19.497880 master-1 kubenswrapper[4771]: I1011 10:36:19.497740 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:19.497880 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:19.497880 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:19.497880 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:19.497880 master-1 kubenswrapper[4771]: I1011 10:36:19.497868 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:20.497290 master-1 kubenswrapper[4771]: I1011 10:36:20.497183 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:20.497290 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:20.497290 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:20.497290 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:20.497758 master-1 kubenswrapper[4771]: I1011 10:36:20.497296 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:21.497651 master-1 kubenswrapper[4771]: I1011 10:36:21.497528 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:21.497651 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:21.497651 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:21.497651 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:21.498646 master-1 kubenswrapper[4771]: I1011 10:36:21.497642 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:22.496875 master-1 kubenswrapper[4771]: I1011 10:36:22.496808 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:22.496875 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:22.496875 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:22.496875 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:22.497489 master-1 kubenswrapper[4771]: I1011 10:36:22.496903 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:22.545139 master-1 kubenswrapper[4771]: I1011 10:36:22.545051 4771 kubelet.go:2431] "SyncLoop REMOVE" source="file" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:36:22.546140 master-1 kubenswrapper[4771]: I1011 10:36:22.545530 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler" containerID="cri-o://63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f" gracePeriod=30 Oct 11 10:36:22.546140 master-1 kubenswrapper[4771]: I1011 10:36:22.545620 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler-recovery-controller" containerID="cri-o://e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316" gracePeriod=30 Oct 11 10:36:22.546140 master-1 kubenswrapper[4771]: I1011 10:36:22.545623 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler-cert-syncer" containerID="cri-o://e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5" gracePeriod=30 Oct 11 10:36:22.547377 master-1 kubenswrapper[4771]: I1011 10:36:22.546626 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:36:22.547377 master-1 kubenswrapper[4771]: E1011 10:36:22.547250 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a61df698d34d049669621b2249bfe758" containerName="wait-for-host-port" Oct 11 10:36:22.547377 master-1 kubenswrapper[4771]: I1011 10:36:22.547279 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61df698d34d049669621b2249bfe758" containerName="wait-for-host-port" Oct 11 10:36:22.547377 master-1 kubenswrapper[4771]: E1011 10:36:22.547306 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler-recovery-controller" Oct 11 10:36:22.547377 master-1 kubenswrapper[4771]: I1011 10:36:22.547323 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler-recovery-controller" Oct 11 10:36:22.547377 master-1 kubenswrapper[4771]: E1011 10:36:22.547340 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler" Oct 11 10:36:22.547377 master-1 kubenswrapper[4771]: I1011 10:36:22.547377 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler" Oct 11 10:36:22.547804 master-1 kubenswrapper[4771]: E1011 10:36:22.547394 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler-cert-syncer" Oct 11 10:36:22.547804 master-1 kubenswrapper[4771]: I1011 10:36:22.547407 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler-cert-syncer" Oct 11 10:36:22.547804 master-1 kubenswrapper[4771]: I1011 10:36:22.547640 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler-recovery-controller" Oct 11 10:36:22.547804 master-1 kubenswrapper[4771]: I1011 10:36:22.547710 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler" Oct 11 10:36:22.547804 master-1 kubenswrapper[4771]: I1011 10:36:22.547733 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a61df698d34d049669621b2249bfe758" containerName="kube-scheduler-cert-syncer" Oct 11 10:36:22.595045 master-1 kubenswrapper[4771]: I1011 10:36:22.594962 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/1ffd3b5548bcf48fce7bfb9a8c802165-cert-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"1ffd3b5548bcf48fce7bfb9a8c802165\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:22.595304 master-1 kubenswrapper[4771]: I1011 10:36:22.595246 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/1ffd3b5548bcf48fce7bfb9a8c802165-resource-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"1ffd3b5548bcf48fce7bfb9a8c802165\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:22.695889 master-1 kubenswrapper[4771]: I1011 10:36:22.695795 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/1ffd3b5548bcf48fce7bfb9a8c802165-resource-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"1ffd3b5548bcf48fce7bfb9a8c802165\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:22.695889 master-1 kubenswrapper[4771]: I1011 10:36:22.695883 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/1ffd3b5548bcf48fce7bfb9a8c802165-cert-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"1ffd3b5548bcf48fce7bfb9a8c802165\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:22.696195 master-1 kubenswrapper[4771]: I1011 10:36:22.695969 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/1ffd3b5548bcf48fce7bfb9a8c802165-cert-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"1ffd3b5548bcf48fce7bfb9a8c802165\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:22.696195 master-1 kubenswrapper[4771]: I1011 10:36:22.695977 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/1ffd3b5548bcf48fce7bfb9a8c802165-resource-dir\") pod \"openshift-kube-scheduler-master-1\" (UID: \"1ffd3b5548bcf48fce7bfb9a8c802165\") " pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: I1011 10:36:23.244955 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:36:23.245042 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:36:23.247629 master-1 kubenswrapper[4771]: I1011 10:36:23.245042 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:23.497259 master-1 kubenswrapper[4771]: I1011 10:36:23.497068 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:23.497259 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:23.497259 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:23.497259 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:23.497259 master-1 kubenswrapper[4771]: I1011 10:36:23.497161 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:23.609101 master-1 kubenswrapper[4771]: I1011 10:36:23.609029 4771 generic.go:334] "Generic (PLEG): container finished" podID="67e39e90-67d5-40f4-ad76-1b32adf359ed" containerID="4ac39222fba40ff7cbe78740b5c6cfd319b2ad66eef840556f4373378718527a" exitCode=0 Oct 11 10:36:23.609986 master-1 kubenswrapper[4771]: I1011 10:36:23.609177 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-6-master-1" event={"ID":"67e39e90-67d5-40f4-ad76-1b32adf359ed","Type":"ContainerDied","Data":"4ac39222fba40ff7cbe78740b5c6cfd319b2ad66eef840556f4373378718527a"} Oct 11 10:36:23.612215 master-1 kubenswrapper[4771]: I1011 10:36:23.612159 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_a61df698d34d049669621b2249bfe758/kube-scheduler-cert-syncer/0.log" Oct 11 10:36:23.613340 master-1 kubenswrapper[4771]: I1011 10:36:23.613295 4771 generic.go:334] "Generic (PLEG): container finished" podID="a61df698d34d049669621b2249bfe758" containerID="e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316" exitCode=0 Oct 11 10:36:23.613340 master-1 kubenswrapper[4771]: I1011 10:36:23.613331 4771 generic.go:334] "Generic (PLEG): container finished" podID="a61df698d34d049669621b2249bfe758" containerID="e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5" exitCode=2 Oct 11 10:36:23.615746 master-1 kubenswrapper[4771]: I1011 10:36:23.615682 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="a61df698d34d049669621b2249bfe758" podUID="1ffd3b5548bcf48fce7bfb9a8c802165" Oct 11 10:36:24.496935 master-1 kubenswrapper[4771]: I1011 10:36:24.496835 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:24.496935 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:24.496935 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:24.496935 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:24.497475 master-1 kubenswrapper[4771]: I1011 10:36:24.496950 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:24.975941 master-1 kubenswrapper[4771]: I1011 10:36:24.975875 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:36:25.042349 master-1 kubenswrapper[4771]: I1011 10:36:25.042262 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-kubelet-dir\") pod \"67e39e90-67d5-40f4-ad76-1b32adf359ed\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " Oct 11 10:36:25.042349 master-1 kubenswrapper[4771]: I1011 10:36:25.042333 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-var-lock\") pod \"67e39e90-67d5-40f4-ad76-1b32adf359ed\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " Oct 11 10:36:25.042789 master-1 kubenswrapper[4771]: I1011 10:36:25.042399 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "67e39e90-67d5-40f4-ad76-1b32adf359ed" (UID: "67e39e90-67d5-40f4-ad76-1b32adf359ed"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:36:25.042789 master-1 kubenswrapper[4771]: I1011 10:36:25.042431 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/67e39e90-67d5-40f4-ad76-1b32adf359ed-kube-api-access\") pod \"67e39e90-67d5-40f4-ad76-1b32adf359ed\" (UID: \"67e39e90-67d5-40f4-ad76-1b32adf359ed\") " Oct 11 10:36:25.042789 master-1 kubenswrapper[4771]: I1011 10:36:25.042465 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-var-lock" (OuterVolumeSpecName: "var-lock") pod "67e39e90-67d5-40f4-ad76-1b32adf359ed" (UID: "67e39e90-67d5-40f4-ad76-1b32adf359ed"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:36:25.042789 master-1 kubenswrapper[4771]: I1011 10:36:25.042710 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:36:25.042789 master-1 kubenswrapper[4771]: I1011 10:36:25.042733 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/67e39e90-67d5-40f4-ad76-1b32adf359ed-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:36:25.047333 master-1 kubenswrapper[4771]: I1011 10:36:25.047250 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e39e90-67d5-40f4-ad76-1b32adf359ed-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "67e39e90-67d5-40f4-ad76-1b32adf359ed" (UID: "67e39e90-67d5-40f4-ad76-1b32adf359ed"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:36:25.145182 master-1 kubenswrapper[4771]: I1011 10:36:25.145041 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/67e39e90-67d5-40f4-ad76-1b32adf359ed-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:36:25.497814 master-1 kubenswrapper[4771]: I1011 10:36:25.497751 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:25.497814 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:25.497814 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:25.497814 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:25.498401 master-1 kubenswrapper[4771]: I1011 10:36:25.498325 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:25.627480 master-1 kubenswrapper[4771]: I1011 10:36:25.627401 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/installer-6-master-1" event={"ID":"67e39e90-67d5-40f4-ad76-1b32adf359ed","Type":"ContainerDied","Data":"c6be9acab2b46a0600e4b835c238bce39535b79bcb5079c1d439519d3a10a7ca"} Oct 11 10:36:25.627480 master-1 kubenswrapper[4771]: I1011 10:36:25.627481 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6be9acab2b46a0600e4b835c238bce39535b79bcb5079c1d439519d3a10a7ca" Oct 11 10:36:25.627896 master-1 kubenswrapper[4771]: I1011 10:36:25.627862 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/installer-6-master-1" Oct 11 10:36:25.906263 master-1 kubenswrapper[4771]: E1011 10:36:25.906034 4771 event.go:359] "Server rejected event (will not retry!)" err="etcdserver: request timed out, possibly due to previous leader failure" event="&Event{ObjectMeta:{redhat-marketplace-9ncpc.186d696f3f80d6d9 openshift-marketplace 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-marketplace,Name:redhat-marketplace-9ncpc,UID:91e987bb-eae2-4f14-809d-1b1141882c7d,APIVersion:v1,ResourceVersion:15448,FieldPath:spec.containers{registry-server},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2ed3a56ac3e411dffa5a6d960e8ab570b62cc00a560c485d3eb5c4eb34c9cc5\" in 475ms (475ms including waiting). Image size: 911296197 bytes.,Source:EventSource{Component:kubelet,Host:master-1,},FirstTimestamp:2025-10-11 10:35:55.899426521 +0000 UTC m=+587.873652962,LastTimestamp:2025-10-11 10:35:55.899426521 +0000 UTC m=+587.873652962,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-1,}" Oct 11 10:36:26.497568 master-1 kubenswrapper[4771]: I1011 10:36:26.497443 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:26.497568 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:26.497568 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:26.497568 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:26.497568 master-1 kubenswrapper[4771]: I1011 10:36:26.497534 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:26.607959 master-1 kubenswrapper[4771]: I1011 10:36:26.607834 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:36:26.608244 master-1 kubenswrapper[4771]: I1011 10:36:26.607945 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:36:27.496702 master-1 kubenswrapper[4771]: I1011 10:36:27.496609 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:27.496702 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:27.496702 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:27.496702 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:27.496702 master-1 kubenswrapper[4771]: I1011 10:36:27.496678 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:28.237959 master-1 kubenswrapper[4771]: I1011 10:36:28.237858 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:36:28.237959 master-1 kubenswrapper[4771]: I1011 10:36:28.237932 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:36:28.238964 master-1 kubenswrapper[4771]: I1011 10:36:28.238612 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_a61df698d34d049669621b2249bfe758/kube-scheduler-cert-syncer/0.log" Oct 11 10:36:28.240054 master-1 kubenswrapper[4771]: I1011 10:36:28.240010 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:28.245791 master-1 kubenswrapper[4771]: I1011 10:36:28.245742 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="a61df698d34d049669621b2249bfe758" podUID="1ffd3b5548bcf48fce7bfb9a8c802165" Oct 11 10:36:28.282613 master-1 kubenswrapper[4771]: I1011 10:36:28.282549 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-cert-dir\") pod \"a61df698d34d049669621b2249bfe758\" (UID: \"a61df698d34d049669621b2249bfe758\") " Oct 11 10:36:28.282789 master-1 kubenswrapper[4771]: I1011 10:36:28.282650 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-cert-dir" (OuterVolumeSpecName: "cert-dir") pod "a61df698d34d049669621b2249bfe758" (UID: "a61df698d34d049669621b2249bfe758"). InnerVolumeSpecName "cert-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:36:28.282789 master-1 kubenswrapper[4771]: I1011 10:36:28.282732 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-resource-dir\") pod \"a61df698d34d049669621b2249bfe758\" (UID: \"a61df698d34d049669621b2249bfe758\") " Oct 11 10:36:28.282933 master-1 kubenswrapper[4771]: I1011 10:36:28.282847 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-resource-dir" (OuterVolumeSpecName: "resource-dir") pod "a61df698d34d049669621b2249bfe758" (UID: "a61df698d34d049669621b2249bfe758"). InnerVolumeSpecName "resource-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:36:28.283041 master-1 kubenswrapper[4771]: I1011 10:36:28.283018 4771 reconciler_common.go:293] "Volume detached for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-cert-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:36:28.283041 master-1 kubenswrapper[4771]: I1011 10:36:28.283037 4771 reconciler_common.go:293] "Volume detached for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/a61df698d34d049669621b2249bfe758-resource-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:36:28.447471 master-1 kubenswrapper[4771]: I1011 10:36:28.447245 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a61df698d34d049669621b2249bfe758" path="/var/lib/kubelet/pods/a61df698d34d049669621b2249bfe758/volumes" Oct 11 10:36:28.498639 master-1 kubenswrapper[4771]: I1011 10:36:28.498544 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:28.498639 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:28.498639 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:28.498639 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:28.498639 master-1 kubenswrapper[4771]: I1011 10:36:28.498633 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:28.653020 master-1 kubenswrapper[4771]: I1011 10:36:28.648860 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_a61df698d34d049669621b2249bfe758/kube-scheduler-cert-syncer/0.log" Oct 11 10:36:28.653020 master-1 kubenswrapper[4771]: I1011 10:36:28.651987 4771 generic.go:334] "Generic (PLEG): container finished" podID="a61df698d34d049669621b2249bfe758" containerID="63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f" exitCode=0 Oct 11 10:36:28.653020 master-1 kubenswrapper[4771]: I1011 10:36:28.652124 4771 scope.go:117] "RemoveContainer" containerID="e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316" Oct 11 10:36:28.653020 master-1 kubenswrapper[4771]: I1011 10:36:28.652659 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:28.658663 master-1 kubenswrapper[4771]: I1011 10:36:28.658591 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="a61df698d34d049669621b2249bfe758" podUID="1ffd3b5548bcf48fce7bfb9a8c802165" Oct 11 10:36:28.708405 master-1 kubenswrapper[4771]: I1011 10:36:28.708177 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="a61df698d34d049669621b2249bfe758" podUID="1ffd3b5548bcf48fce7bfb9a8c802165" Oct 11 10:36:28.714148 master-1 kubenswrapper[4771]: I1011 10:36:28.714087 4771 scope.go:117] "RemoveContainer" containerID="e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5" Oct 11 10:36:28.715687 master-1 kubenswrapper[4771]: I1011 10:36:28.715641 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_34b1362996d1e0c2cea0bee73eb18468/kube-apiserver-cert-syncer/0.log" Oct 11 10:36:28.716911 master-1 kubenswrapper[4771]: I1011 10:36:28.716874 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:36:28.724161 master-1 kubenswrapper[4771]: I1011 10:36:28.724103 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="34b1362996d1e0c2cea0bee73eb18468" podUID="e39186c2ebd02622803bdbec6984de2a" Oct 11 10:36:28.735116 master-1 kubenswrapper[4771]: I1011 10:36:28.735035 4771 scope.go:117] "RemoveContainer" containerID="63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f" Oct 11 10:36:28.751540 master-1 kubenswrapper[4771]: I1011 10:36:28.751387 4771 scope.go:117] "RemoveContainer" containerID="0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577" Oct 11 10:36:28.778107 master-1 kubenswrapper[4771]: I1011 10:36:28.778059 4771 scope.go:117] "RemoveContainer" containerID="e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316" Oct 11 10:36:28.778612 master-1 kubenswrapper[4771]: E1011 10:36:28.778580 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316\": container with ID starting with e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316 not found: ID does not exist" containerID="e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316" Oct 11 10:36:28.778661 master-1 kubenswrapper[4771]: I1011 10:36:28.778611 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316"} err="failed to get container status \"e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316\": rpc error: code = NotFound desc = could not find container \"e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316\": container with ID starting with e3bd4833cb6b364aa83fca88897d32509fa085f61942c4a3cb75cdf814b22316 not found: ID does not exist" Oct 11 10:36:28.778661 master-1 kubenswrapper[4771]: I1011 10:36:28.778635 4771 scope.go:117] "RemoveContainer" containerID="e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5" Oct 11 10:36:28.779030 master-1 kubenswrapper[4771]: E1011 10:36:28.778983 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5\": container with ID starting with e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5 not found: ID does not exist" containerID="e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5" Oct 11 10:36:28.779066 master-1 kubenswrapper[4771]: I1011 10:36:28.779031 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5"} err="failed to get container status \"e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5\": rpc error: code = NotFound desc = could not find container \"e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5\": container with ID starting with e12640b361e2ef73c6e289951cab68eae36f0b9bc81be5fd9209771124b251a5 not found: ID does not exist" Oct 11 10:36:28.779066 master-1 kubenswrapper[4771]: I1011 10:36:28.779054 4771 scope.go:117] "RemoveContainer" containerID="63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f" Oct 11 10:36:28.779413 master-1 kubenswrapper[4771]: E1011 10:36:28.779392 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f\": container with ID starting with 63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f not found: ID does not exist" containerID="63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f" Oct 11 10:36:28.779456 master-1 kubenswrapper[4771]: I1011 10:36:28.779413 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f"} err="failed to get container status \"63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f\": rpc error: code = NotFound desc = could not find container \"63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f\": container with ID starting with 63c473df6fa732d6511618b295787205edc98e4025bcc6c14cf2f92361ce263f not found: ID does not exist" Oct 11 10:36:28.779456 master-1 kubenswrapper[4771]: I1011 10:36:28.779427 4771 scope.go:117] "RemoveContainer" containerID="0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577" Oct 11 10:36:28.779765 master-1 kubenswrapper[4771]: E1011 10:36:28.779728 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577\": container with ID starting with 0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577 not found: ID does not exist" containerID="0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577" Oct 11 10:36:28.779799 master-1 kubenswrapper[4771]: I1011 10:36:28.779770 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577"} err="failed to get container status \"0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577\": rpc error: code = NotFound desc = could not find container \"0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577\": container with ID starting with 0ce25f4770e41c16a3e6b3f94195d1bb5c5e1e16d7e54fe07f63d76f33baa577 not found: ID does not exist" Oct 11 10:36:28.788979 master-1 kubenswrapper[4771]: I1011 10:36:28.788956 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-resource-dir\") pod \"34b1362996d1e0c2cea0bee73eb18468\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " Oct 11 10:36:28.789036 master-1 kubenswrapper[4771]: I1011 10:36:28.788985 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-audit-dir\") pod \"34b1362996d1e0c2cea0bee73eb18468\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " Oct 11 10:36:28.789036 master-1 kubenswrapper[4771]: I1011 10:36:28.789032 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-cert-dir\") pod \"34b1362996d1e0c2cea0bee73eb18468\" (UID: \"34b1362996d1e0c2cea0bee73eb18468\") " Oct 11 10:36:28.789163 master-1 kubenswrapper[4771]: I1011 10:36:28.789129 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-resource-dir" (OuterVolumeSpecName: "resource-dir") pod "34b1362996d1e0c2cea0bee73eb18468" (UID: "34b1362996d1e0c2cea0bee73eb18468"). InnerVolumeSpecName "resource-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:36:28.789198 master-1 kubenswrapper[4771]: I1011 10:36:28.789175 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-cert-dir" (OuterVolumeSpecName: "cert-dir") pod "34b1362996d1e0c2cea0bee73eb18468" (UID: "34b1362996d1e0c2cea0bee73eb18468"). InnerVolumeSpecName "cert-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:36:28.789240 master-1 kubenswrapper[4771]: I1011 10:36:28.789224 4771 reconciler_common.go:293] "Volume detached for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-resource-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:36:28.789449 master-1 kubenswrapper[4771]: I1011 10:36:28.789312 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "34b1362996d1e0c2cea0bee73eb18468" (UID: "34b1362996d1e0c2cea0bee73eb18468"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:36:28.891560 master-1 kubenswrapper[4771]: I1011 10:36:28.891441 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:36:28.891560 master-1 kubenswrapper[4771]: I1011 10:36:28.891536 4771 reconciler_common.go:293] "Volume detached for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/34b1362996d1e0c2cea0bee73eb18468-cert-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:36:29.438137 master-1 kubenswrapper[4771]: I1011 10:36:29.437715 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:36:29.438763 master-1 kubenswrapper[4771]: I1011 10:36:29.438162 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:36:29.497458 master-1 kubenswrapper[4771]: I1011 10:36:29.497334 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:29.497458 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:29.497458 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:29.497458 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:29.497458 master-1 kubenswrapper[4771]: I1011 10:36:29.497409 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:29.663722 master-1 kubenswrapper[4771]: I1011 10:36:29.663626 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_34b1362996d1e0c2cea0bee73eb18468/kube-apiserver-cert-syncer/0.log" Oct 11 10:36:29.664910 master-1 kubenswrapper[4771]: I1011 10:36:29.664843 4771 generic.go:334] "Generic (PLEG): container finished" podID="34b1362996d1e0c2cea0bee73eb18468" containerID="7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea" exitCode=0 Oct 11 10:36:29.665052 master-1 kubenswrapper[4771]: I1011 10:36:29.664990 4771 scope.go:117] "RemoveContainer" containerID="49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd" Oct 11 10:36:29.665156 master-1 kubenswrapper[4771]: I1011 10:36:29.665086 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:36:29.670586 master-1 kubenswrapper[4771]: I1011 10:36:29.670516 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="34b1362996d1e0c2cea0bee73eb18468" podUID="e39186c2ebd02622803bdbec6984de2a" Oct 11 10:36:29.685160 master-1 kubenswrapper[4771]: I1011 10:36:29.685072 4771 scope.go:117] "RemoveContainer" containerID="3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7" Oct 11 10:36:29.698378 master-1 kubenswrapper[4771]: I1011 10:36:29.698287 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="34b1362996d1e0c2cea0bee73eb18468" podUID="e39186c2ebd02622803bdbec6984de2a" Oct 11 10:36:29.707224 master-1 kubenswrapper[4771]: I1011 10:36:29.707104 4771 scope.go:117] "RemoveContainer" containerID="d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10" Oct 11 10:36:29.725274 master-1 kubenswrapper[4771]: I1011 10:36:29.725195 4771 scope.go:117] "RemoveContainer" containerID="c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1" Oct 11 10:36:29.748648 master-1 kubenswrapper[4771]: I1011 10:36:29.748585 4771 scope.go:117] "RemoveContainer" containerID="7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea" Oct 11 10:36:29.768526 master-1 kubenswrapper[4771]: I1011 10:36:29.768464 4771 scope.go:117] "RemoveContainer" containerID="e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296" Oct 11 10:36:29.801562 master-1 kubenswrapper[4771]: I1011 10:36:29.801291 4771 scope.go:117] "RemoveContainer" containerID="49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd" Oct 11 10:36:29.802322 master-1 kubenswrapper[4771]: E1011 10:36:29.802246 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd\": container with ID starting with 49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd not found: ID does not exist" containerID="49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd" Oct 11 10:36:29.802480 master-1 kubenswrapper[4771]: I1011 10:36:29.802318 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd"} err="failed to get container status \"49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd\": rpc error: code = NotFound desc = could not find container \"49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd\": container with ID starting with 49c30f2556445e43240415750853bc066c7b179aee60a314f7f90ecd36a50dcd not found: ID does not exist" Oct 11 10:36:29.802480 master-1 kubenswrapper[4771]: I1011 10:36:29.802408 4771 scope.go:117] "RemoveContainer" containerID="3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7" Oct 11 10:36:29.803094 master-1 kubenswrapper[4771]: E1011 10:36:29.803038 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7\": container with ID starting with 3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7 not found: ID does not exist" containerID="3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7" Oct 11 10:36:29.803185 master-1 kubenswrapper[4771]: I1011 10:36:29.803081 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7"} err="failed to get container status \"3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7\": rpc error: code = NotFound desc = could not find container \"3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7\": container with ID starting with 3a652daa6b9bec9ce58b6d5f79f895564498416d8415b206a1a52c5a9a98d3f7 not found: ID does not exist" Oct 11 10:36:29.803185 master-1 kubenswrapper[4771]: I1011 10:36:29.803115 4771 scope.go:117] "RemoveContainer" containerID="d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10" Oct 11 10:36:29.803658 master-1 kubenswrapper[4771]: E1011 10:36:29.803595 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10\": container with ID starting with d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10 not found: ID does not exist" containerID="d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10" Oct 11 10:36:29.803761 master-1 kubenswrapper[4771]: I1011 10:36:29.803653 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10"} err="failed to get container status \"d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10\": rpc error: code = NotFound desc = could not find container \"d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10\": container with ID starting with d197bce3e8ba0a7f6aff105b6e86788c609756474629f070cf3ae2b1f7ecea10 not found: ID does not exist" Oct 11 10:36:29.803761 master-1 kubenswrapper[4771]: I1011 10:36:29.803685 4771 scope.go:117] "RemoveContainer" containerID="c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1" Oct 11 10:36:29.804220 master-1 kubenswrapper[4771]: E1011 10:36:29.804166 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1\": container with ID starting with c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1 not found: ID does not exist" containerID="c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1" Oct 11 10:36:29.804302 master-1 kubenswrapper[4771]: I1011 10:36:29.804210 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1"} err="failed to get container status \"c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1\": rpc error: code = NotFound desc = could not find container \"c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1\": container with ID starting with c610027bfa053a5744033e9524cf5428c551fc36958f998db58017c8d204f5c1 not found: ID does not exist" Oct 11 10:36:29.804302 master-1 kubenswrapper[4771]: I1011 10:36:29.804238 4771 scope.go:117] "RemoveContainer" containerID="7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea" Oct 11 10:36:29.804710 master-1 kubenswrapper[4771]: E1011 10:36:29.804633 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea\": container with ID starting with 7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea not found: ID does not exist" containerID="7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea" Oct 11 10:36:29.804794 master-1 kubenswrapper[4771]: I1011 10:36:29.804711 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea"} err="failed to get container status \"7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea\": rpc error: code = NotFound desc = could not find container \"7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea\": container with ID starting with 7b6b0f0ebd2652368dabe4dd6da18ce3abd22cb6b9eb7a0abf73685386aeadea not found: ID does not exist" Oct 11 10:36:29.804794 master-1 kubenswrapper[4771]: I1011 10:36:29.804763 4771 scope.go:117] "RemoveContainer" containerID="e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296" Oct 11 10:36:29.805253 master-1 kubenswrapper[4771]: E1011 10:36:29.805202 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296\": container with ID starting with e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296 not found: ID does not exist" containerID="e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296" Oct 11 10:36:29.805344 master-1 kubenswrapper[4771]: I1011 10:36:29.805249 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296"} err="failed to get container status \"e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296\": rpc error: code = NotFound desc = could not find container \"e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296\": container with ID starting with e822f71821ab6da8ae22657d298be7ecdc1b3a32a18978e721ac45edbf111296 not found: ID does not exist" Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: I1011 10:36:30.248656 4771 patch_prober.go:28] interesting pod/metrics-server-65d86dff78-bg7lk container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: [+]metric-storage-ready ok Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: [+]metric-informer-sync ok Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: [+]metadata-informer-sync ok Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:36:30.248788 master-1 kubenswrapper[4771]: I1011 10:36:30.248778 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:30.447381 master-1 kubenswrapper[4771]: I1011 10:36:30.447238 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34b1362996d1e0c2cea0bee73eb18468" path="/var/lib/kubelet/pods/34b1362996d1e0c2cea0bee73eb18468/volumes" Oct 11 10:36:30.497873 master-1 kubenswrapper[4771]: I1011 10:36:30.497761 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:30.497873 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:30.497873 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:30.497873 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:30.497873 master-1 kubenswrapper[4771]: I1011 10:36:30.497854 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:31.497409 master-1 kubenswrapper[4771]: I1011 10:36:31.497255 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:31.497409 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:31.497409 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:31.497409 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:31.498502 master-1 kubenswrapper[4771]: I1011 10:36:31.497414 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:31.607255 master-1 kubenswrapper[4771]: I1011 10:36:31.607131 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:36:31.607255 master-1 kubenswrapper[4771]: I1011 10:36:31.607239 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:36:32.497487 master-1 kubenswrapper[4771]: I1011 10:36:32.497376 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:32.497487 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:32.497487 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:32.497487 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:32.498542 master-1 kubenswrapper[4771]: I1011 10:36:32.497566 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:33.237809 master-1 kubenswrapper[4771]: I1011 10:36:33.237736 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:36:33.238120 master-1 kubenswrapper[4771]: I1011 10:36:33.237818 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:36:33.498164 master-1 kubenswrapper[4771]: I1011 10:36:33.497997 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:33.498164 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:33.498164 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:33.498164 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:33.499512 master-1 kubenswrapper[4771]: I1011 10:36:33.499458 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:34.496959 master-1 kubenswrapper[4771]: I1011 10:36:34.496889 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:34.496959 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:34.496959 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:34.496959 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:34.497289 master-1 kubenswrapper[4771]: I1011 10:36:34.496982 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:35.497430 master-1 kubenswrapper[4771]: I1011 10:36:35.497320 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:35.497430 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:35.497430 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:35.497430 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:35.498431 master-1 kubenswrapper[4771]: I1011 10:36:35.497436 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:36.437207 master-1 kubenswrapper[4771]: I1011 10:36:36.437135 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:36.458647 master-1 kubenswrapper[4771]: I1011 10:36:36.458580 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:36:36.458647 master-1 kubenswrapper[4771]: I1011 10:36:36.458640 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:36:36.485082 master-1 kubenswrapper[4771]: I1011 10:36:36.485022 4771 kubelet.go:1914] "Deleted mirror pod because it is outdated" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:36.498010 master-1 kubenswrapper[4771]: I1011 10:36:36.497915 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:36.498010 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:36.498010 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:36.498010 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:36.499005 master-1 kubenswrapper[4771]: I1011 10:36:36.498012 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:36.503856 master-1 kubenswrapper[4771]: I1011 10:36:36.503818 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:36:36.530883 master-1 kubenswrapper[4771]: W1011 10:36:36.530791 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffd3b5548bcf48fce7bfb9a8c802165.slice/crio-9d69cbb2721062172f20487d13f1ddcc1efa6d2ad3a2f2fe34e0e943d4e5aff7 WatchSource:0}: Error finding container 9d69cbb2721062172f20487d13f1ddcc1efa6d2ad3a2f2fe34e0e943d4e5aff7: Status 404 returned error can't find the container with id 9d69cbb2721062172f20487d13f1ddcc1efa6d2ad3a2f2fe34e0e943d4e5aff7 Oct 11 10:36:36.606931 master-1 kubenswrapper[4771]: I1011 10:36:36.606835 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:36:36.606931 master-1 kubenswrapper[4771]: I1011 10:36:36.606924 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:36:36.607214 master-1 kubenswrapper[4771]: I1011 10:36:36.607032 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:36:36.607585 master-1 kubenswrapper[4771]: I1011 10:36:36.607533 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:36:36.607674 master-1 kubenswrapper[4771]: I1011 10:36:36.607581 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:36:36.707531 master-1 kubenswrapper[4771]: I1011 10:36:36.707468 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"1ffd3b5548bcf48fce7bfb9a8c802165","Type":"ContainerStarted","Data":"9d69cbb2721062172f20487d13f1ddcc1efa6d2ad3a2f2fe34e0e943d4e5aff7"} Oct 11 10:36:37.437134 master-1 kubenswrapper[4771]: I1011 10:36:37.437008 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:36:37.459332 master-1 kubenswrapper[4771]: I1011 10:36:37.459249 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:36:37.459332 master-1 kubenswrapper[4771]: I1011 10:36:37.459307 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:36:37.486504 master-1 kubenswrapper[4771]: I1011 10:36:37.486414 4771 kubelet.go:1914] "Deleted mirror pod because it is outdated" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:36:37.497339 master-1 kubenswrapper[4771]: I1011 10:36:37.497213 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:37.497339 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:37.497339 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:37.497339 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:37.497339 master-1 kubenswrapper[4771]: I1011 10:36:37.497332 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:37.509455 master-1 kubenswrapper[4771]: I1011 10:36:37.509328 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:36:37.537104 master-1 kubenswrapper[4771]: W1011 10:36:37.537007 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39186c2ebd02622803bdbec6984de2a.slice/crio-ff12d9d351de2bbe6eed10390bc259efe990212b31b335f9dbf3e11e9545f55b WatchSource:0}: Error finding container ff12d9d351de2bbe6eed10390bc259efe990212b31b335f9dbf3e11e9545f55b: Status 404 returned error can't find the container with id ff12d9d351de2bbe6eed10390bc259efe990212b31b335f9dbf3e11e9545f55b Oct 11 10:36:37.718971 master-1 kubenswrapper[4771]: I1011 10:36:37.718894 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"e39186c2ebd02622803bdbec6984de2a","Type":"ContainerStarted","Data":"ff12d9d351de2bbe6eed10390bc259efe990212b31b335f9dbf3e11e9545f55b"} Oct 11 10:36:37.720534 master-1 kubenswrapper[4771]: I1011 10:36:37.720492 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"1ffd3b5548bcf48fce7bfb9a8c802165","Type":"ContainerStarted","Data":"e1bceca0463892708755bd3c74753a682b8d9cb1da4a4db3b72edc17dbc8cf99"} Oct 11 10:36:37.720987 master-1 kubenswrapper[4771]: I1011 10:36:37.720929 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:36:37.720987 master-1 kubenswrapper[4771]: I1011 10:36:37.720980 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:36:38.238936 master-1 kubenswrapper[4771]: I1011 10:36:38.238812 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:36:38.238936 master-1 kubenswrapper[4771]: I1011 10:36:38.238901 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:36:38.496987 master-1 kubenswrapper[4771]: I1011 10:36:38.496652 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:38.496987 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:38.496987 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:38.496987 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:38.496987 master-1 kubenswrapper[4771]: I1011 10:36:38.496707 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:38.727263 master-1 kubenswrapper[4771]: I1011 10:36:38.727202 4771 generic.go:334] "Generic (PLEG): container finished" podID="e39186c2ebd02622803bdbec6984de2a" containerID="70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912" exitCode=0 Oct 11 10:36:38.727858 master-1 kubenswrapper[4771]: I1011 10:36:38.727279 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"e39186c2ebd02622803bdbec6984de2a","Type":"ContainerDied","Data":"70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912"} Oct 11 10:36:38.727858 master-1 kubenswrapper[4771]: I1011 10:36:38.727608 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:36:38.727858 master-1 kubenswrapper[4771]: I1011 10:36:38.727626 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:36:38.727858 master-1 kubenswrapper[4771]: I1011 10:36:38.727704 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:36:38.727858 master-1 kubenswrapper[4771]: I1011 10:36:38.727728 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:36:39.497546 master-1 kubenswrapper[4771]: I1011 10:36:39.497435 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:39.497546 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:39.497546 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:39.497546 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:39.497546 master-1 kubenswrapper[4771]: I1011 10:36:39.497537 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:39.745206 master-1 kubenswrapper[4771]: I1011 10:36:39.742923 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"e39186c2ebd02622803bdbec6984de2a","Type":"ContainerStarted","Data":"7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109"} Oct 11 10:36:39.745206 master-1 kubenswrapper[4771]: I1011 10:36:39.742978 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"e39186c2ebd02622803bdbec6984de2a","Type":"ContainerStarted","Data":"0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd"} Oct 11 10:36:39.745206 master-1 kubenswrapper[4771]: I1011 10:36:39.742991 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"e39186c2ebd02622803bdbec6984de2a","Type":"ContainerStarted","Data":"be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a"} Oct 11 10:36:40.496490 master-1 kubenswrapper[4771]: I1011 10:36:40.496380 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:40.496490 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:40.496490 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:40.496490 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:40.496490 master-1 kubenswrapper[4771]: I1011 10:36:40.496456 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:40.750469 master-1 kubenswrapper[4771]: I1011 10:36:40.750362 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"e39186c2ebd02622803bdbec6984de2a","Type":"ContainerStarted","Data":"50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07"} Oct 11 10:36:40.750469 master-1 kubenswrapper[4771]: I1011 10:36:40.750414 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"e39186c2ebd02622803bdbec6984de2a","Type":"ContainerStarted","Data":"637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4"} Oct 11 10:36:40.750931 master-1 kubenswrapper[4771]: I1011 10:36:40.750534 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:36:40.750931 master-1 kubenswrapper[4771]: I1011 10:36:40.750627 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:36:40.750931 master-1 kubenswrapper[4771]: I1011 10:36:40.750646 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:36:41.497041 master-1 kubenswrapper[4771]: I1011 10:36:41.496942 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:41.497041 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:41.497041 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:41.497041 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:41.497500 master-1 kubenswrapper[4771]: I1011 10:36:41.497050 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:41.607243 master-1 kubenswrapper[4771]: I1011 10:36:41.607169 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:36:41.607505 master-1 kubenswrapper[4771]: I1011 10:36:41.607292 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:36:41.758392 master-1 kubenswrapper[4771]: I1011 10:36:41.758178 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:36:41.758392 master-1 kubenswrapper[4771]: I1011 10:36:41.758235 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:36:42.458086 master-1 kubenswrapper[4771]: I1011 10:36:42.457979 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="1ffd3b5548bcf48fce7bfb9a8c802165" podUID="72ed0d6c-89fd-4c0b-bd7d-af241a60bf0c" Oct 11 10:36:42.482916 master-1 kubenswrapper[4771]: I1011 10:36:42.482820 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="e39186c2ebd02622803bdbec6984de2a" podUID="233d76fa-d8e2-41eb-9272-6cdd0056b793" Oct 11 10:36:42.497184 master-1 kubenswrapper[4771]: I1011 10:36:42.497115 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:42.497184 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:42.497184 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:42.497184 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:42.497431 master-1 kubenswrapper[4771]: I1011 10:36:42.497236 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:43.245207 master-1 kubenswrapper[4771]: I1011 10:36:43.245144 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:36:43.496887 master-1 kubenswrapper[4771]: I1011 10:36:43.496697 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:43.496887 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:43.496887 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:43.496887 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:43.496887 master-1 kubenswrapper[4771]: I1011 10:36:43.496790 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:44.497802 master-1 kubenswrapper[4771]: I1011 10:36:44.497707 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:44.497802 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:44.497802 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:44.497802 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:44.498974 master-1 kubenswrapper[4771]: I1011 10:36:44.497816 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:45.497422 master-1 kubenswrapper[4771]: I1011 10:36:45.497302 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:45.497422 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:45.497422 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:45.497422 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:45.497869 master-1 kubenswrapper[4771]: I1011 10:36:45.497422 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:46.497015 master-1 kubenswrapper[4771]: I1011 10:36:46.496936 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:46.497015 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:46.497015 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:46.497015 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:46.497543 master-1 kubenswrapper[4771]: I1011 10:36:46.497019 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:46.608485 master-1 kubenswrapper[4771]: I1011 10:36:46.608390 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:36:46.609060 master-1 kubenswrapper[4771]: I1011 10:36:46.608503 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:36:47.497305 master-1 kubenswrapper[4771]: I1011 10:36:47.497214 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:47.497305 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:47.497305 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:47.497305 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:47.497811 master-1 kubenswrapper[4771]: I1011 10:36:47.497308 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:48.498346 master-1 kubenswrapper[4771]: I1011 10:36:48.498247 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:48.498346 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:48.498346 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:48.498346 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:48.498985 master-1 kubenswrapper[4771]: I1011 10:36:48.498383 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:48.801696 master-1 kubenswrapper[4771]: I1011 10:36:48.801627 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-node-identity_network-node-identity-sk5cm_0e03dddd-4197-40ae-91f1-7e83f90dbd58/approver/0.log" Oct 11 10:36:48.802426 master-1 kubenswrapper[4771]: I1011 10:36:48.802324 4771 generic.go:334] "Generic (PLEG): container finished" podID="0e03dddd-4197-40ae-91f1-7e83f90dbd58" containerID="c93dfaf9a8b9fa7850e31e158a74ae1fbf85ec41153c0883cb5064b10872afdb" exitCode=1 Oct 11 10:36:48.802596 master-1 kubenswrapper[4771]: I1011 10:36:48.802422 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-node-identity/network-node-identity-sk5cm" event={"ID":"0e03dddd-4197-40ae-91f1-7e83f90dbd58","Type":"ContainerDied","Data":"c93dfaf9a8b9fa7850e31e158a74ae1fbf85ec41153c0883cb5064b10872afdb"} Oct 11 10:36:48.803328 master-1 kubenswrapper[4771]: I1011 10:36:48.803251 4771 scope.go:117] "RemoveContainer" containerID="c93dfaf9a8b9fa7850e31e158a74ae1fbf85ec41153c0883cb5064b10872afdb" Oct 11 10:36:49.497490 master-1 kubenswrapper[4771]: I1011 10:36:49.497347 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:49.497490 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:49.497490 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:49.497490 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:49.497942 master-1 kubenswrapper[4771]: I1011 10:36:49.497494 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:49.811652 master-1 kubenswrapper[4771]: I1011 10:36:49.811568 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-node-identity_network-node-identity-sk5cm_0e03dddd-4197-40ae-91f1-7e83f90dbd58/approver/0.log" Oct 11 10:36:49.812762 master-1 kubenswrapper[4771]: I1011 10:36:49.812199 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-node-identity/network-node-identity-sk5cm" event={"ID":"0e03dddd-4197-40ae-91f1-7e83f90dbd58","Type":"ContainerStarted","Data":"bace159805a524fcefe768a06f0db4deab6f2bcec258b041da2198e3ef2b3ee4"} Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: I1011 10:36:50.245713 4771 patch_prober.go:28] interesting pod/metrics-server-65d86dff78-bg7lk container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: [+]metric-storage-ready ok Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: [+]metric-informer-sync ok Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: [+]metadata-informer-sync ok Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:36:50.245900 master-1 kubenswrapper[4771]: I1011 10:36:50.245864 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:50.246865 master-1 kubenswrapper[4771]: I1011 10:36:50.246069 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:36:50.497295 master-1 kubenswrapper[4771]: I1011 10:36:50.497085 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:50.497295 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:50.497295 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:50.497295 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:50.497295 master-1 kubenswrapper[4771]: I1011 10:36:50.497181 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:51.497082 master-1 kubenswrapper[4771]: I1011 10:36:51.497020 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:51.497082 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:51.497082 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:51.497082 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:51.497082 master-1 kubenswrapper[4771]: I1011 10:36:51.497097 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:51.607853 master-1 kubenswrapper[4771]: I1011 10:36:51.607745 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:36:51.608145 master-1 kubenswrapper[4771]: I1011 10:36:51.607849 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:36:52.497318 master-1 kubenswrapper[4771]: I1011 10:36:52.497227 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:52.497318 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:52.497318 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:52.497318 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:52.498815 master-1 kubenswrapper[4771]: I1011 10:36:52.498758 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:53.497533 master-1 kubenswrapper[4771]: I1011 10:36:53.497475 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:53.497533 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:53.497533 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:53.497533 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:53.497533 master-1 kubenswrapper[4771]: I1011 10:36:53.497534 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:54.497870 master-1 kubenswrapper[4771]: I1011 10:36:54.497774 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:54.497870 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:54.497870 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:54.497870 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:54.498905 master-1 kubenswrapper[4771]: I1011 10:36:54.497875 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:55.501859 master-1 kubenswrapper[4771]: I1011 10:36:55.501798 4771 patch_prober.go:28] interesting pod/router-default-5ddb89f76-z5t6x container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-http failed: reason withheld Oct 11 10:36:55.501859 master-1 kubenswrapper[4771]: [-]has-synced failed: reason withheld Oct 11 10:36:55.501859 master-1 kubenswrapper[4771]: [+]process-running ok Oct 11 10:36:55.501859 master-1 kubenswrapper[4771]: healthz check failed Oct 11 10:36:55.503471 master-1 kubenswrapper[4771]: I1011 10:36:55.501887 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" podUID="04cd4a19-2532-43d1-9144-1f59d9e52d19" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:36:56.498395 master-1 kubenswrapper[4771]: I1011 10:36:56.498266 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:36:56.502496 master-1 kubenswrapper[4771]: I1011 10:36:56.502421 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5ddb89f76-z5t6x" Oct 11 10:36:56.607601 master-1 kubenswrapper[4771]: I1011 10:36:56.607501 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:36:56.607601 master-1 kubenswrapper[4771]: I1011 10:36:56.607583 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:36:59.433205 master-1 kubenswrapper[4771]: I1011 10:36:59.433096 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:36:59.434228 master-1 kubenswrapper[4771]: E1011 10:36:59.433409 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:39:01.433335615 +0000 UTC m=+773.407562086 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:36:59.535084 master-1 kubenswrapper[4771]: I1011 10:36:59.534985 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:36:59.535400 master-1 kubenswrapper[4771]: E1011 10:36:59.535247 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:39:01.535215177 +0000 UTC m=+773.509441708 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:36:59.949906 master-1 kubenswrapper[4771]: E1011 10:36:59.949825 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" podUID="d7647696-42d9-4dd9-bc3b-a4d52a42cf9a" Oct 11 10:36:59.950255 master-1 kubenswrapper[4771]: E1011 10:36:59.949918 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" podUID="6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b" Oct 11 10:37:00.887508 master-1 kubenswrapper[4771]: I1011 10:37:00.887430 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:37:00.887508 master-1 kubenswrapper[4771]: I1011 10:37:00.887470 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:37:01.607387 master-1 kubenswrapper[4771]: I1011 10:37:01.607290 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:37:01.607387 master-1 kubenswrapper[4771]: I1011 10:37:01.607389 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:37:06.607344 master-1 kubenswrapper[4771]: I1011 10:37:06.607206 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:37:06.607344 master-1 kubenswrapper[4771]: I1011 10:37:06.607309 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:37:06.928422 master-1 kubenswrapper[4771]: I1011 10:37:06.928223 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_1ffd3b5548bcf48fce7bfb9a8c802165/wait-for-host-port/0.log" Oct 11 10:37:06.928422 master-1 kubenswrapper[4771]: I1011 10:37:06.928320 4771 generic.go:334] "Generic (PLEG): container finished" podID="1ffd3b5548bcf48fce7bfb9a8c802165" containerID="e1bceca0463892708755bd3c74753a682b8d9cb1da4a4db3b72edc17dbc8cf99" exitCode=124 Oct 11 10:37:06.928728 master-1 kubenswrapper[4771]: I1011 10:37:06.928419 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"1ffd3b5548bcf48fce7bfb9a8c802165","Type":"ContainerDied","Data":"e1bceca0463892708755bd3c74753a682b8d9cb1da4a4db3b72edc17dbc8cf99"} Oct 11 10:37:06.928918 master-1 kubenswrapper[4771]: I1011 10:37:06.928870 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:06.928918 master-1 kubenswrapper[4771]: I1011 10:37:06.928907 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:06.936645 master-1 kubenswrapper[4771]: I1011 10:37:06.935875 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="1ffd3b5548bcf48fce7bfb9a8c802165" podUID="72ed0d6c-89fd-4c0b-bd7d-af241a60bf0c" Oct 11 10:37:07.939529 master-1 kubenswrapper[4771]: I1011 10:37:07.939448 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_1ffd3b5548bcf48fce7bfb9a8c802165/wait-for-host-port/0.log" Oct 11 10:37:07.939529 master-1 kubenswrapper[4771]: I1011 10:37:07.939520 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"1ffd3b5548bcf48fce7bfb9a8c802165","Type":"ContainerStarted","Data":"0d52bdbb99c295f13f83ea3eb7c6fa80b331e9642a12b706a9065cf6d85ba5e0"} Oct 11 10:37:07.940460 master-1 kubenswrapper[4771]: I1011 10:37:07.939840 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:07.940460 master-1 kubenswrapper[4771]: I1011 10:37:07.939860 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:07.947578 master-1 kubenswrapper[4771]: I1011 10:37:07.947508 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="1ffd3b5548bcf48fce7bfb9a8c802165" podUID="72ed0d6c-89fd-4c0b-bd7d-af241a60bf0c" Oct 11 10:37:10.248017 master-1 kubenswrapper[4771]: I1011 10:37:10.247940 4771 patch_prober.go:28] interesting pod/metrics-server-65d86dff78-bg7lk container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:37:10.248017 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:37:10.248017 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:37:10.248017 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:37:10.248017 master-1 kubenswrapper[4771]: [+]metric-storage-ready ok Oct 11 10:37:10.248017 master-1 kubenswrapper[4771]: [+]metric-informer-sync ok Oct 11 10:37:10.248017 master-1 kubenswrapper[4771]: [+]metadata-informer-sync ok Oct 11 10:37:10.248017 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:37:10.248017 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:37:10.248986 master-1 kubenswrapper[4771]: I1011 10:37:10.248046 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:37:11.607922 master-1 kubenswrapper[4771]: I1011 10:37:11.607841 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:37:11.608752 master-1 kubenswrapper[4771]: I1011 10:37:11.607960 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:37:16.607909 master-1 kubenswrapper[4771]: I1011 10:37:16.607806 4771 patch_prober.go:28] interesting pod/openshift-kube-scheduler-guard-master-1 container/guard namespace/openshift-kube-scheduler: Readiness probe status=failure output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" start-of-body= Oct 11 10:37:16.608610 master-1 kubenswrapper[4771]: I1011 10:37:16.607908 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" podUID="bdeaf49a-fbf2-4e26-88cf-10e723bbdfbe" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10259/healthz\": dial tcp 192.168.34.11:10259: connect: connection refused" Oct 11 10:37:20.016488 master-1 kubenswrapper[4771]: I1011 10:37:20.016312 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-scheduler_openshift-kube-scheduler-master-1_1ffd3b5548bcf48fce7bfb9a8c802165/wait-for-host-port/0.log" Oct 11 10:37:20.017320 master-1 kubenswrapper[4771]: I1011 10:37:20.016486 4771 generic.go:334] "Generic (PLEG): container finished" podID="1ffd3b5548bcf48fce7bfb9a8c802165" containerID="0d52bdbb99c295f13f83ea3eb7c6fa80b331e9642a12b706a9065cf6d85ba5e0" exitCode=0 Oct 11 10:37:20.017320 master-1 kubenswrapper[4771]: I1011 10:37:20.016550 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"1ffd3b5548bcf48fce7bfb9a8c802165","Type":"ContainerDied","Data":"0d52bdbb99c295f13f83ea3eb7c6fa80b331e9642a12b706a9065cf6d85ba5e0"} Oct 11 10:37:20.017320 master-1 kubenswrapper[4771]: I1011 10:37:20.016667 4771 scope.go:117] "RemoveContainer" containerID="e1bceca0463892708755bd3c74753a682b8d9cb1da4a4db3b72edc17dbc8cf99" Oct 11 10:37:20.017320 master-1 kubenswrapper[4771]: I1011 10:37:20.017152 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:20.017320 master-1 kubenswrapper[4771]: I1011 10:37:20.017196 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:20.022972 master-1 kubenswrapper[4771]: I1011 10:37:20.022863 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="1ffd3b5548bcf48fce7bfb9a8c802165" podUID="72ed0d6c-89fd-4c0b-bd7d-af241a60bf0c" Oct 11 10:37:20.024046 master-1 kubenswrapper[4771]: I1011 10:37:20.023974 4771 scope.go:117] "RemoveContainer" containerID="e1bceca0463892708755bd3c74753a682b8d9cb1da4a4db3b72edc17dbc8cf99" Oct 11 10:37:20.053683 master-1 kubenswrapper[4771]: E1011 10:37:20.053575 4771 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_wait-for-host-port_openshift-kube-scheduler-master-1_openshift-kube-scheduler_1ffd3b5548bcf48fce7bfb9a8c802165_0 in pod sandbox 9d69cbb2721062172f20487d13f1ddcc1efa6d2ad3a2f2fe34e0e943d4e5aff7 from index: no such id: 'e1bceca0463892708755bd3c74753a682b8d9cb1da4a4db3b72edc17dbc8cf99'" containerID="e1bceca0463892708755bd3c74753a682b8d9cb1da4a4db3b72edc17dbc8cf99" Oct 11 10:37:20.053846 master-1 kubenswrapper[4771]: I1011 10:37:20.053666 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bceca0463892708755bd3c74753a682b8d9cb1da4a4db3b72edc17dbc8cf99"} err="rpc error: code = Unknown desc = failed to delete container k8s_wait-for-host-port_openshift-kube-scheduler-master-1_openshift-kube-scheduler_1ffd3b5548bcf48fce7bfb9a8c802165_0 in pod sandbox 9d69cbb2721062172f20487d13f1ddcc1efa6d2ad3a2f2fe34e0e943d4e5aff7 from index: no such id: 'e1bceca0463892708755bd3c74753a682b8d9cb1da4a4db3b72edc17dbc8cf99'" Oct 11 10:37:21.047343 master-1 kubenswrapper[4771]: I1011 10:37:21.047249 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"1ffd3b5548bcf48fce7bfb9a8c802165","Type":"ContainerStarted","Data":"e74bc6e341be990d641b3208e5db3f1986459a1e52462f9ffe91cdc733fb24ba"} Oct 11 10:37:21.047343 master-1 kubenswrapper[4771]: I1011 10:37:21.047335 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"1ffd3b5548bcf48fce7bfb9a8c802165","Type":"ContainerStarted","Data":"20eb65db094b78f38b3f02d62c1011c3f80b66cf8dad3e6c74359f3d5455aa20"} Oct 11 10:37:21.048426 master-1 kubenswrapper[4771]: I1011 10:37:21.047394 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" event={"ID":"1ffd3b5548bcf48fce7bfb9a8c802165","Type":"ContainerStarted","Data":"fac2308db6f6681cb5064427b63efb1dc9e4c6c67dbcb6bdd98a8dcd331c1383"} Oct 11 10:37:21.048426 master-1 kubenswrapper[4771]: I1011 10:37:21.047823 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:21.048426 master-1 kubenswrapper[4771]: I1011 10:37:21.047884 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:21.048426 master-1 kubenswrapper[4771]: I1011 10:37:21.048236 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:37:21.057394 master-1 kubenswrapper[4771]: I1011 10:37:21.057277 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="1ffd3b5548bcf48fce7bfb9a8c802165" podUID="72ed0d6c-89fd-4c0b-bd7d-af241a60bf0c" Oct 11 10:37:21.616957 master-1 kubenswrapper[4771]: I1011 10:37:21.616891 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-scheduler/openshift-kube-scheduler-guard-master-1" Oct 11 10:37:22.053245 master-1 kubenswrapper[4771]: I1011 10:37:22.053192 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:22.053245 master-1 kubenswrapper[4771]: I1011 10:37:22.053226 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:22.056695 master-1 kubenswrapper[4771]: I1011 10:37:22.056661 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="1ffd3b5548bcf48fce7bfb9a8c802165" podUID="72ed0d6c-89fd-4c0b-bd7d-af241a60bf0c" Oct 11 10:37:28.096986 master-1 kubenswrapper[4771]: I1011 10:37:28.096919 4771 generic.go:334] "Generic (PLEG): container finished" podID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerID="d71774e5747fba198d1f1c685867c43372766be8110c50262b34cb5aee247b7d" exitCode=0 Oct 11 10:37:28.096986 master-1 kubenswrapper[4771]: I1011 10:37:28.096974 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" event={"ID":"daf74cdb-6bdb-465a-8e3e-194e8868570f","Type":"ContainerDied","Data":"d71774e5747fba198d1f1c685867c43372766be8110c50262b34cb5aee247b7d"} Oct 11 10:37:28.267610 master-1 kubenswrapper[4771]: I1011 10:37:28.267523 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:37:28.346154 master-1 kubenswrapper[4771]: I1011 10:37:28.345955 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/daf74cdb-6bdb-465a-8e3e-194e8868570f-audit-log\") pod \"daf74cdb-6bdb-465a-8e3e-194e8868570f\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " Oct 11 10:37:28.346154 master-1 kubenswrapper[4771]: I1011 10:37:28.346054 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-client-ca-bundle\") pod \"daf74cdb-6bdb-465a-8e3e-194e8868570f\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " Oct 11 10:37:28.346154 master-1 kubenswrapper[4771]: I1011 10:37:28.346101 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-metrics-server-audit-profiles\") pod \"daf74cdb-6bdb-465a-8e3e-194e8868570f\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " Oct 11 10:37:28.346154 master-1 kubenswrapper[4771]: I1011 10:37:28.346130 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-server-tls\") pod \"daf74cdb-6bdb-465a-8e3e-194e8868570f\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " Oct 11 10:37:28.346154 master-1 kubenswrapper[4771]: I1011 10:37:28.346165 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-configmap-kubelet-serving-ca-bundle\") pod \"daf74cdb-6bdb-465a-8e3e-194e8868570f\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " Oct 11 10:37:28.346578 master-1 kubenswrapper[4771]: I1011 10:37:28.346204 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s2hx\" (UniqueName: \"kubernetes.io/projected/daf74cdb-6bdb-465a-8e3e-194e8868570f-kube-api-access-5s2hx\") pod \"daf74cdb-6bdb-465a-8e3e-194e8868570f\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " Oct 11 10:37:28.346578 master-1 kubenswrapper[4771]: I1011 10:37:28.346234 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-client-certs\") pod \"daf74cdb-6bdb-465a-8e3e-194e8868570f\" (UID: \"daf74cdb-6bdb-465a-8e3e-194e8868570f\") " Oct 11 10:37:28.347274 master-1 kubenswrapper[4771]: I1011 10:37:28.347183 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "daf74cdb-6bdb-465a-8e3e-194e8868570f" (UID: "daf74cdb-6bdb-465a-8e3e-194e8868570f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:37:28.347547 master-1 kubenswrapper[4771]: I1011 10:37:28.347459 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-metrics-server-audit-profiles" (OuterVolumeSpecName: "metrics-server-audit-profiles") pod "daf74cdb-6bdb-465a-8e3e-194e8868570f" (UID: "daf74cdb-6bdb-465a-8e3e-194e8868570f"). InnerVolumeSpecName "metrics-server-audit-profiles". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:37:28.347606 master-1 kubenswrapper[4771]: I1011 10:37:28.347496 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf74cdb-6bdb-465a-8e3e-194e8868570f-audit-log" (OuterVolumeSpecName: "audit-log") pod "daf74cdb-6bdb-465a-8e3e-194e8868570f" (UID: "daf74cdb-6bdb-465a-8e3e-194e8868570f"). InnerVolumeSpecName "audit-log". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:37:28.350747 master-1 kubenswrapper[4771]: I1011 10:37:28.350679 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-client-ca-bundle" (OuterVolumeSpecName: "client-ca-bundle") pod "daf74cdb-6bdb-465a-8e3e-194e8868570f" (UID: "daf74cdb-6bdb-465a-8e3e-194e8868570f"). InnerVolumeSpecName "client-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:37:28.351190 master-1 kubenswrapper[4771]: I1011 10:37:28.351123 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf74cdb-6bdb-465a-8e3e-194e8868570f-kube-api-access-5s2hx" (OuterVolumeSpecName: "kube-api-access-5s2hx") pod "daf74cdb-6bdb-465a-8e3e-194e8868570f" (UID: "daf74cdb-6bdb-465a-8e3e-194e8868570f"). InnerVolumeSpecName "kube-api-access-5s2hx". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:37:28.351686 master-1 kubenswrapper[4771]: I1011 10:37:28.351643 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-server-tls" (OuterVolumeSpecName: "secret-metrics-server-tls") pod "daf74cdb-6bdb-465a-8e3e-194e8868570f" (UID: "daf74cdb-6bdb-465a-8e3e-194e8868570f"). InnerVolumeSpecName "secret-metrics-server-tls". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:37:28.352233 master-1 kubenswrapper[4771]: I1011 10:37:28.352176 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "daf74cdb-6bdb-465a-8e3e-194e8868570f" (UID: "daf74cdb-6bdb-465a-8e3e-194e8868570f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:37:28.448059 master-1 kubenswrapper[4771]: I1011 10:37:28.447971 4771 reconciler_common.go:293] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-client-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:28.448059 master-1 kubenswrapper[4771]: I1011 10:37:28.448018 4771 reconciler_common.go:293] "Volume detached for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/daf74cdb-6bdb-465a-8e3e-194e8868570f-audit-log\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:28.448059 master-1 kubenswrapper[4771]: I1011 10:37:28.448032 4771 reconciler_common.go:293] "Volume detached for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-client-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:28.448059 master-1 kubenswrapper[4771]: I1011 10:37:28.448045 4771 reconciler_common.go:293] "Volume detached for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-metrics-server-audit-profiles\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:28.448059 master-1 kubenswrapper[4771]: I1011 10:37:28.448062 4771 reconciler_common.go:293] "Volume detached for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/daf74cdb-6bdb-465a-8e3e-194e8868570f-secret-metrics-server-tls\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:28.448059 master-1 kubenswrapper[4771]: I1011 10:37:28.448075 4771 reconciler_common.go:293] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf74cdb-6bdb-465a-8e3e-194e8868570f-configmap-kubelet-serving-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:28.448698 master-1 kubenswrapper[4771]: I1011 10:37:28.448089 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-5s2hx\" (UniqueName: \"kubernetes.io/projected/daf74cdb-6bdb-465a-8e3e-194e8868570f-kube-api-access-5s2hx\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:29.104262 master-1 kubenswrapper[4771]: I1011 10:37:29.104200 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" event={"ID":"daf74cdb-6bdb-465a-8e3e-194e8868570f","Type":"ContainerDied","Data":"307edc8bf8db53981b4988030525d3bc29e6569573860e0ae13cb952073e6408"} Oct 11 10:37:29.104262 master-1 kubenswrapper[4771]: I1011 10:37:29.104269 4771 scope.go:117] "RemoveContainer" containerID="d71774e5747fba198d1f1c685867c43372766be8110c50262b34cb5aee247b7d" Oct 11 10:37:29.105187 master-1 kubenswrapper[4771]: I1011 10:37:29.104294 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65d86dff78-bg7lk" Oct 11 10:37:36.751962 master-1 kubenswrapper[4771]: I1011 10:37:36.751869 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-marketplace-9ncpc" podStartSLOduration=102.220313808 podStartE2EDuration="1m44.75184485s" podCreationTimestamp="2025-10-11 10:35:52 +0000 UTC" firstStartedPulling="2025-10-11 10:35:53.367879779 +0000 UTC m=+585.342106230" lastFinishedPulling="2025-10-11 10:35:55.899410831 +0000 UTC m=+587.873637272" observedRunningTime="2025-10-11 10:35:58.036101539 +0000 UTC m=+590.010328020" watchObservedRunningTime="2025-10-11 10:37:36.75184485 +0000 UTC m=+688.726071331" Oct 11 10:37:36.771089 master-1 kubenswrapper[4771]: I1011 10:37:36.770979 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-operators-plxkp" podStartSLOduration=102.358018525 podStartE2EDuration="1m44.770954956s" podCreationTimestamp="2025-10-11 10:35:52 +0000 UTC" firstStartedPulling="2025-10-11 10:35:54.404160024 +0000 UTC m=+586.378386495" lastFinishedPulling="2025-10-11 10:35:56.817096435 +0000 UTC m=+588.791322926" observedRunningTime="2025-10-11 10:35:58.054774232 +0000 UTC m=+590.029000673" watchObservedRunningTime="2025-10-11 10:37:36.770954956 +0000 UTC m=+688.745181407" Oct 11 10:37:36.861424 master-1 kubenswrapper[4771]: I1011 10:37:36.861335 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:37:36.863553 master-1 kubenswrapper[4771]: I1011 10:37:36.863487 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:36.863693 master-1 kubenswrapper[4771]: I1011 10:37:36.863677 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podUID="70960866-cb10-49c0-b28d-18f7fa34215c" Oct 11 10:37:36.868519 master-1 kubenswrapper[4771]: I1011 10:37:36.868455 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:37:36.878122 master-1 kubenswrapper[4771]: I1011 10:37:36.878053 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" oldPodUID="1ffd3b5548bcf48fce7bfb9a8c802165" podUID="72ed0d6c-89fd-4c0b-bd7d-af241a60bf0c" Oct 11 10:37:36.880902 master-1 kubenswrapper[4771]: I1011 10:37:36.880847 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/openshift-kube-scheduler-master-1"] Oct 11 10:37:36.891711 master-1 kubenswrapper[4771]: I1011 10:37:36.891636 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:37:36.897739 master-1 kubenswrapper[4771]: I1011 10:37:36.895480 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:37:36.897739 master-1 kubenswrapper[4771]: I1011 10:37:36.895882 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:37:36.897739 master-1 kubenswrapper[4771]: I1011 10:37:36.896025 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:37:36.897739 master-1 kubenswrapper[4771]: I1011 10:37:36.896050 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="1087abee-42bf-483a-b523-13b9169544e5" Oct 11 10:37:36.898693 master-1 kubenswrapper[4771]: I1011 10:37:36.898647 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/installer-5-master-1"] Oct 11 10:37:36.898949 master-1 kubenswrapper[4771]: E1011 10:37:36.898926 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" Oct 11 10:37:36.898949 master-1 kubenswrapper[4771]: I1011 10:37:36.898946 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" Oct 11 10:37:36.899132 master-1 kubenswrapper[4771]: E1011 10:37:36.898967 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="67e39e90-67d5-40f4-ad76-1b32adf359ed" containerName="installer" Oct 11 10:37:36.899132 master-1 kubenswrapper[4771]: I1011 10:37:36.898976 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e39e90-67d5-40f4-ad76-1b32adf359ed" containerName="installer" Oct 11 10:37:36.899132 master-1 kubenswrapper[4771]: I1011 10:37:36.899106 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" containerName="metrics-server" Oct 11 10:37:36.899132 master-1 kubenswrapper[4771]: I1011 10:37:36.899126 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="67e39e90-67d5-40f4-ad76-1b32adf359ed" containerName="installer" Oct 11 10:37:36.899720 master-1 kubenswrapper[4771]: I1011 10:37:36.899688 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:36.904444 master-1 kubenswrapper[4771]: I1011 10:37:36.904385 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:37:36.904754 master-1 kubenswrapper[4771]: I1011 10:37:36.904708 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-kube-controller-manager"/"installer-sa-dockercfg-djvlq" Oct 11 10:37:36.935843 master-1 kubenswrapper[4771]: I1011 10:37:36.935722 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd"] Oct 11 10:37:36.937203 master-1 kubenswrapper[4771]: I1011 10:37:36.937168 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/metrics-server-65d86dff78-bg7lk"] Oct 11 10:37:36.937375 master-1 kubenswrapper[4771]: I1011 10:37:36.937334 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:36.938749 master-1 kubenswrapper[4771]: I1011 10:37:36.938668 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/metrics-server-65d86dff78-bg7lk"] Oct 11 10:37:36.941550 master-1 kubenswrapper[4771]: I1011 10:37:36.941344 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"metrics-client-certs" Oct 11 10:37:36.941896 master-1 kubenswrapper[4771]: I1011 10:37:36.941850 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"metrics-server-audit-profiles" Oct 11 10:37:36.942079 master-1 kubenswrapper[4771]: I1011 10:37:36.942049 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"kubelet-serving-ca-bundle" Oct 11 10:37:36.942634 master-1 kubenswrapper[4771]: I1011 10:37:36.942614 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"metrics-server-tls" Oct 11 10:37:36.943071 master-1 kubenswrapper[4771]: I1011 10:37:36.943051 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"metrics-server-2ocquro0n92lc" Oct 11 10:37:36.947702 master-1 kubenswrapper[4771]: I1011 10:37:36.947659 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-authentication/oauth-openshift-68fb97bcc4-g7k57"] Oct 11 10:37:36.948482 master-1 kubenswrapper[4771]: I1011 10:37:36.948459 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:36.952950 master-1 kubenswrapper[4771]: I1011 10:37:36.952913 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-authentication"/"v4-0-config-system-cliconfig" Oct 11 10:37:36.953912 master-1 kubenswrapper[4771]: I1011 10:37:36.953888 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-authentication"/"v4-0-config-system-service-ca" Oct 11 10:37:36.954225 master-1 kubenswrapper[4771]: I1011 10:37:36.954203 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-authentication"/"v4-0-config-system-serving-cert" Oct 11 10:37:36.954524 master-1 kubenswrapper[4771]: I1011 10:37:36.954500 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-authentication"/"v4-0-config-user-template-provider-selection" Oct 11 10:37:36.954744 master-1 kubenswrapper[4771]: I1011 10:37:36.954723 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-authentication"/"v4-0-config-system-router-certs" Oct 11 10:37:36.954853 master-1 kubenswrapper[4771]: I1011 10:37:36.954824 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-authentication"/"v4-0-config-system-session" Oct 11 10:37:36.955017 master-1 kubenswrapper[4771]: I1011 10:37:36.954995 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-authentication"/"v4-0-config-user-template-login" Oct 11 10:37:36.955189 master-1 kubenswrapper[4771]: I1011 10:37:36.955105 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-authentication"/"oauth-openshift-dockercfg-279hr" Oct 11 10:37:36.956037 master-1 kubenswrapper[4771]: I1011 10:37:36.956007 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-var-lock\") pod \"installer-5-master-1\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:36.956135 master-1 kubenswrapper[4771]: I1011 10:37:36.956107 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kube-api-access\") pod \"installer-5-master-1\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:36.956186 master-1 kubenswrapper[4771]: I1011 10:37:36.956161 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:36.957063 master-1 kubenswrapper[4771]: I1011 10:37:36.957026 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-authentication"/"kube-root-ca.crt" Oct 11 10:37:36.958776 master-1 kubenswrapper[4771]: I1011 10:37:36.957550 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-authentication"/"v4-0-config-user-template-error" Oct 11 10:37:36.958776 master-1 kubenswrapper[4771]: I1011 10:37:36.957778 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-authentication"/"audit" Oct 11 10:37:36.958776 master-1 kubenswrapper[4771]: I1011 10:37:36.957833 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-authentication"/"openshift-service-ca.crt" Oct 11 10:37:36.961972 master-1 kubenswrapper[4771]: I1011 10:37:36.961897 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/kube-apiserver-master-1" podStartSLOduration=59.961877294 podStartE2EDuration="59.961877294s" podCreationTimestamp="2025-10-11 10:36:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:37:36.96000762 +0000 UTC m=+688.934234091" watchObservedRunningTime="2025-10-11 10:37:36.961877294 +0000 UTC m=+688.936103735" Oct 11 10:37:36.968024 master-1 kubenswrapper[4771]: I1011 10:37:36.967979 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-authentication"/"v4-0-config-system-trusted-ca-bundle" Oct 11 10:37:36.976029 master-1 kubenswrapper[4771]: I1011 10:37:36.975978 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-authentication"/"v4-0-config-system-ocp-branding-template" Oct 11 10:37:37.057613 master-1 kubenswrapper[4771]: I1011 10:37:37.057564 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56hdc\" (UniqueName: \"kubernetes.io/projected/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-kube-api-access-56hdc\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.057831 master-1 kubenswrapper[4771]: I1011 10:37:37.057637 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dd18178e-3cb1-41de-8866-913f8f23d90d-metrics-server-audit-profiles\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.057831 master-1 kubenswrapper[4771]: I1011 10:37:37.057671 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-user-template-provider-selection\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-provider-selection\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.057831 master-1 kubenswrapper[4771]: I1011 10:37:37.057705 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:37.057831 master-1 kubenswrapper[4771]: I1011 10:37:37.057729 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-policies\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.057831 master-1 kubenswrapper[4771]: I1011 10:37:37.057755 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dd18178e-3cb1-41de-8866-913f8f23d90d-audit-log\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.057831 master-1 kubenswrapper[4771]: I1011 10:37:37.057787 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dd18178e-3cb1-41de-8866-913f8f23d90d-client-ca-bundle\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.057831 master-1 kubenswrapper[4771]: I1011 10:37:37.057821 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-dir\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058044 master-1 kubenswrapper[4771]: I1011 10:37:37.057845 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-session\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-session\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058044 master-1 kubenswrapper[4771]: I1011 10:37:37.057851 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:37.058044 master-1 kubenswrapper[4771]: I1011 10:37:37.057869 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-service-ca\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058044 master-1 kubenswrapper[4771]: I1011 10:37:37.057981 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-ocp-branding-template\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-ocp-branding-template\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058489 master-1 kubenswrapper[4771]: I1011 10:37:37.058464 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-router-certs\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-router-certs\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058565 master-1 kubenswrapper[4771]: I1011 10:37:37.058515 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-user-template-error\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-error\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058565 master-1 kubenswrapper[4771]: I1011 10:37:37.058547 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd18178e-3cb1-41de-8866-913f8f23d90d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.058663 master-1 kubenswrapper[4771]: I1011 10:37:37.058582 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd18178e-3cb1-41de-8866-913f8f23d90d-secret-metrics-client-certs\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.058663 master-1 kubenswrapper[4771]: I1011 10:37:37.058605 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-trusted-ca-bundle\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058663 master-1 kubenswrapper[4771]: I1011 10:37:37.058631 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bvtq\" (UniqueName: \"kubernetes.io/projected/dd18178e-3cb1-41de-8866-913f8f23d90d-kube-api-access-5bvtq\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.058663 master-1 kubenswrapper[4771]: I1011 10:37:37.058656 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-serving-cert\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058933 master-1 kubenswrapper[4771]: I1011 10:37:37.058686 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-var-lock\") pod \"installer-5-master-1\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:37.058933 master-1 kubenswrapper[4771]: I1011 10:37:37.058725 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-user-template-login\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-login\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058933 master-1 kubenswrapper[4771]: I1011 10:37:37.058751 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-cliconfig\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-cliconfig\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.058933 master-1 kubenswrapper[4771]: I1011 10:37:37.058778 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dd18178e-3cb1-41de-8866-913f8f23d90d-secret-metrics-server-tls\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.058933 master-1 kubenswrapper[4771]: I1011 10:37:37.058804 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kube-api-access\") pod \"installer-5-master-1\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:37.059176 master-1 kubenswrapper[4771]: I1011 10:37:37.059078 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-var-lock\") pod \"installer-5-master-1\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:37.159858 master-1 kubenswrapper[4771]: I1011 10:37:37.159801 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-ocp-branding-template\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-ocp-branding-template\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160128 master-1 kubenswrapper[4771]: I1011 10:37:37.159883 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-router-certs\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-router-certs\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160128 master-1 kubenswrapper[4771]: I1011 10:37:37.159946 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-user-template-error\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-error\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160128 master-1 kubenswrapper[4771]: I1011 10:37:37.159972 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd18178e-3cb1-41de-8866-913f8f23d90d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.160128 master-1 kubenswrapper[4771]: I1011 10:37:37.159994 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd18178e-3cb1-41de-8866-913f8f23d90d-secret-metrics-client-certs\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.160128 master-1 kubenswrapper[4771]: I1011 10:37:37.160023 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-trusted-ca-bundle\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160128 master-1 kubenswrapper[4771]: I1011 10:37:37.160050 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvtq\" (UniqueName: \"kubernetes.io/projected/dd18178e-3cb1-41de-8866-913f8f23d90d-kube-api-access-5bvtq\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.160128 master-1 kubenswrapper[4771]: I1011 10:37:37.160077 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-serving-cert\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160128 master-1 kubenswrapper[4771]: I1011 10:37:37.160118 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-user-template-login\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-login\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160143 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-cliconfig\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-cliconfig\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160194 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dd18178e-3cb1-41de-8866-913f8f23d90d-secret-metrics-server-tls\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160250 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-56hdc\" (UniqueName: \"kubernetes.io/projected/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-kube-api-access-56hdc\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160282 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dd18178e-3cb1-41de-8866-913f8f23d90d-metrics-server-audit-profiles\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160310 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-user-template-provider-selection\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-provider-selection\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160341 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-policies\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160386 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dd18178e-3cb1-41de-8866-913f8f23d90d-audit-log\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160424 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dd18178e-3cb1-41de-8866-913f8f23d90d-client-ca-bundle\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160453 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-dir\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160477 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-session\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-session\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.160585 master-1 kubenswrapper[4771]: I1011 10:37:37.160499 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-service-ca\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.161501 master-1 kubenswrapper[4771]: I1011 10:37:37.161303 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-service-ca\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.161501 master-1 kubenswrapper[4771]: I1011 10:37:37.161430 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd18178e-3cb1-41de-8866-913f8f23d90d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.161622 master-1 kubenswrapper[4771]: I1011 10:37:37.161537 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-dir\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.161675 master-1 kubenswrapper[4771]: I1011 10:37:37.161652 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dd18178e-3cb1-41de-8866-913f8f23d90d-audit-log\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.162973 master-1 kubenswrapper[4771]: I1011 10:37:37.162909 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dd18178e-3cb1-41de-8866-913f8f23d90d-metrics-server-audit-profiles\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.164130 master-1 kubenswrapper[4771]: I1011 10:37:37.163693 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-policies\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.164130 master-1 kubenswrapper[4771]: I1011 10:37:37.164039 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-cliconfig\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-cliconfig\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.164447 master-1 kubenswrapper[4771]: I1011 10:37:37.164401 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-user-template-error\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-error\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.164447 master-1 kubenswrapper[4771]: I1011 10:37:37.164436 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-trusted-ca-bundle\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.164998 master-1 kubenswrapper[4771]: I1011 10:37:37.164956 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dd18178e-3cb1-41de-8866-913f8f23d90d-secret-metrics-server-tls\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.165144 master-1 kubenswrapper[4771]: I1011 10:37:37.165082 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd18178e-3cb1-41de-8866-913f8f23d90d-secret-metrics-client-certs\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.165838 master-1 kubenswrapper[4771]: I1011 10:37:37.165795 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-router-certs\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-router-certs\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.166479 master-1 kubenswrapper[4771]: I1011 10:37:37.166447 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-ocp-branding-template\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-ocp-branding-template\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.167041 master-1 kubenswrapper[4771]: I1011 10:37:37.167007 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-serving-cert\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.167566 master-1 kubenswrapper[4771]: I1011 10:37:37.167517 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kube-api-access\") pod \"installer-5-master-1\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:37.167799 master-1 kubenswrapper[4771]: I1011 10:37:37.167758 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dd18178e-3cb1-41de-8866-913f8f23d90d-client-ca-bundle\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.168081 master-1 kubenswrapper[4771]: I1011 10:37:37.168016 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-session\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-session\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.168257 master-1 kubenswrapper[4771]: I1011 10:37:37.168216 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-user-template-provider-selection\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-provider-selection\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.184465 master-1 kubenswrapper[4771]: I1011 10:37:37.182469 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-user-template-login\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-login\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.222881 master-1 kubenswrapper[4771]: I1011 10:37:37.222649 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:37:37.227793 master-1 kubenswrapper[4771]: I1011 10:37:37.227752 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-56hdc\" (UniqueName: \"kubernetes.io/projected/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-kube-api-access-56hdc\") pod \"oauth-openshift-68fb97bcc4-g7k57\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.233128 master-1 kubenswrapper[4771]: I1011 10:37:37.233065 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvtq\" (UniqueName: \"kubernetes.io/projected/dd18178e-3cb1-41de-8866-913f8f23d90d-kube-api-access-5bvtq\") pod \"metrics-server-7d46fcc5c6-bhfmd\" (UID: \"dd18178e-3cb1-41de-8866-913f8f23d90d\") " pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.257226 master-1 kubenswrapper[4771]: I1011 10:37:37.257139 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:37.268293 master-1 kubenswrapper[4771]: I1011 10:37:37.268209 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:37.336265 master-1 kubenswrapper[4771]: I1011 10:37:37.335415 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" podStartSLOduration=61.335393282 podStartE2EDuration="1m1.335393282s" podCreationTimestamp="2025-10-11 10:36:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:37:37.330978235 +0000 UTC m=+689.305204686" watchObservedRunningTime="2025-10-11 10:37:37.335393282 +0000 UTC m=+689.309619743" Oct 11 10:37:37.511268 master-1 kubenswrapper[4771]: I1011 10:37:37.510563 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:37:37.511497 master-1 kubenswrapper[4771]: I1011 10:37:37.511413 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:37:37.520556 master-1 kubenswrapper[4771]: I1011 10:37:37.520501 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:37:37.864516 master-1 kubenswrapper[4771]: I1011 10:37:37.864464 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp"] Oct 11 10:37:37.865113 master-1 kubenswrapper[4771]: I1011 10:37:37.865094 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" Oct 11 10:37:37.868763 master-1 kubenswrapper[4771]: I1011 10:37:37.868260 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"default-dockercfg-65wvh" Oct 11 10:37:37.869278 master-1 kubenswrapper[4771]: I1011 10:37:37.869129 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"monitoring-plugin-cert" Oct 11 10:37:37.888494 master-1 kubenswrapper[4771]: I1011 10:37:37.888418 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-6768b5f5f9-r74mm"] Oct 11 10:37:37.889771 master-1 kubenswrapper[4771]: I1011 10:37:37.889719 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:37.893675 master-1 kubenswrapper[4771]: I1011 10:37:37.893639 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console-operator"/"console-operator-config" Oct 11 10:37:37.893948 master-1 kubenswrapper[4771]: I1011 10:37:37.893915 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console-operator"/"serving-cert" Oct 11 10:37:37.894222 master-1 kubenswrapper[4771]: I1011 10:37:37.894194 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console-operator"/"kube-root-ca.crt" Oct 11 10:37:37.894432 master-1 kubenswrapper[4771]: I1011 10:37:37.894412 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console-operator"/"console-operator-dockercfg-mqd87" Oct 11 10:37:37.894589 master-1 kubenswrapper[4771]: I1011 10:37:37.894567 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console-operator"/"openshift-service-ca.crt" Oct 11 10:37:37.895325 master-1 kubenswrapper[4771]: I1011 10:37:37.895292 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console-operator"/"trusted-ca" Oct 11 10:37:37.969506 master-1 kubenswrapper[4771]: I1011 10:37:37.969435 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjkd\" (UniqueName: \"kubernetes.io/projected/05330706-8231-4c38-be56-416f243992c3-kube-api-access-2tjkd\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:37.969506 master-1 kubenswrapper[4771]: I1011 10:37:37.969506 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d472d171-a5c8-4c71-9d31-7ec0aa3a6db9-monitoring-plugin-cert\") pod \"monitoring-plugin-578f8b47b8-tljlp\" (UID: \"d472d171-a5c8-4c71-9d31-7ec0aa3a6db9\") " pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" Oct 11 10:37:37.969766 master-1 kubenswrapper[4771]: I1011 10:37:37.969535 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05330706-8231-4c38-be56-416f243992c3-serving-cert\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:37.969766 master-1 kubenswrapper[4771]: I1011 10:37:37.969592 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05330706-8231-4c38-be56-416f243992c3-trusted-ca\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:37.969766 master-1 kubenswrapper[4771]: I1011 10:37:37.969649 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05330706-8231-4c38-be56-416f243992c3-config\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.053936 master-1 kubenswrapper[4771]: I1011 10:37:38.053882 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-authentication/oauth-openshift-68fb97bcc4-g7k57"] Oct 11 10:37:38.063074 master-1 kubenswrapper[4771]: I1011 10:37:38.063031 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-6768b5f5f9-r74mm"] Oct 11 10:37:38.066661 master-1 kubenswrapper[4771]: I1011 10:37:38.066591 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-5-master-1"] Oct 11 10:37:38.070797 master-1 kubenswrapper[4771]: I1011 10:37:38.070768 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjkd\" (UniqueName: \"kubernetes.io/projected/05330706-8231-4c38-be56-416f243992c3-kube-api-access-2tjkd\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.070917 master-1 kubenswrapper[4771]: I1011 10:37:38.070904 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d472d171-a5c8-4c71-9d31-7ec0aa3a6db9-monitoring-plugin-cert\") pod \"monitoring-plugin-578f8b47b8-tljlp\" (UID: \"d472d171-a5c8-4c71-9d31-7ec0aa3a6db9\") " pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" Oct 11 10:37:38.070997 master-1 kubenswrapper[4771]: I1011 10:37:38.070984 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05330706-8231-4c38-be56-416f243992c3-serving-cert\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.071097 master-1 kubenswrapper[4771]: I1011 10:37:38.071085 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05330706-8231-4c38-be56-416f243992c3-trusted-ca\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.071179 master-1 kubenswrapper[4771]: I1011 10:37:38.071167 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05330706-8231-4c38-be56-416f243992c3-config\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.072023 master-1 kubenswrapper[4771]: I1011 10:37:38.072006 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05330706-8231-4c38-be56-416f243992c3-config\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.074593 master-1 kubenswrapper[4771]: I1011 10:37:38.074553 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp"] Oct 11 10:37:38.074593 master-1 kubenswrapper[4771]: E1011 10:37:38.074571 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05330706-8231-4c38-be56-416f243992c3-trusted-ca podName:05330706-8231-4c38-be56-416f243992c3 nodeName:}" failed. No retries permitted until 2025-10-11 10:37:38.574556052 +0000 UTC m=+690.548782493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/05330706-8231-4c38-be56-416f243992c3-trusted-ca") pod "console-operator-6768b5f5f9-r74mm" (UID: "05330706-8231-4c38-be56-416f243992c3") : configmap references non-existent config key: ca-bundle.crt Oct 11 10:37:38.077063 master-1 kubenswrapper[4771]: I1011 10:37:38.077043 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d472d171-a5c8-4c71-9d31-7ec0aa3a6db9-monitoring-plugin-cert\") pod \"monitoring-plugin-578f8b47b8-tljlp\" (UID: \"d472d171-a5c8-4c71-9d31-7ec0aa3a6db9\") " pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" Oct 11 10:37:38.079898 master-1 kubenswrapper[4771]: I1011 10:37:38.079880 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05330706-8231-4c38-be56-416f243992c3-serving-cert\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.081459 master-1 kubenswrapper[4771]: I1011 10:37:38.081410 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd"] Oct 11 10:37:38.094145 master-1 kubenswrapper[4771]: I1011 10:37:38.094107 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjkd\" (UniqueName: \"kubernetes.io/projected/05330706-8231-4c38-be56-416f243992c3-kube-api-access-2tjkd\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.178909 master-1 kubenswrapper[4771]: I1011 10:37:38.178825 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" Oct 11 10:37:38.220204 master-1 kubenswrapper[4771]: I1011 10:37:38.219958 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:37:38.446856 master-1 kubenswrapper[4771]: I1011 10:37:38.446732 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf74cdb-6bdb-465a-8e3e-194e8868570f" path="/var/lib/kubelet/pods/daf74cdb-6bdb-465a-8e3e-194e8868570f/volumes" Oct 11 10:37:38.576724 master-1 kubenswrapper[4771]: I1011 10:37:38.576621 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd"] Oct 11 10:37:38.578426 master-1 kubenswrapper[4771]: I1011 10:37:38.578328 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-5-master-1"] Oct 11 10:37:38.583486 master-1 kubenswrapper[4771]: W1011 10:37:38.582095 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd61f55f6_6e03_40ca_aa96_cb6ba21c39b4.slice/crio-d51941eb09c3e4a595d1102ccd7dab8966ed8157ff482e09bff14b3b01ba141c WatchSource:0}: Error finding container d51941eb09c3e4a595d1102ccd7dab8966ed8157ff482e09bff14b3b01ba141c: Status 404 returned error can't find the container with id d51941eb09c3e4a595d1102ccd7dab8966ed8157ff482e09bff14b3b01ba141c Oct 11 10:37:38.586149 master-1 kubenswrapper[4771]: W1011 10:37:38.585895 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd18178e_3cb1_41de_8866_913f8f23d90d.slice/crio-e3c94e0f6f964ee9f94da14b1bdf57bd8704f7e66d4160744c45e29823a9eaca WatchSource:0}: Error finding container e3c94e0f6f964ee9f94da14b1bdf57bd8704f7e66d4160744c45e29823a9eaca: Status 404 returned error can't find the container with id e3c94e0f6f964ee9f94da14b1bdf57bd8704f7e66d4160744c45e29823a9eaca Oct 11 10:37:38.589026 master-1 kubenswrapper[4771]: I1011 10:37:38.588984 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05330706-8231-4c38-be56-416f243992c3-trusted-ca\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.592042 master-1 kubenswrapper[4771]: I1011 10:37:38.591943 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05330706-8231-4c38-be56-416f243992c3-trusted-ca\") pod \"console-operator-6768b5f5f9-r74mm\" (UID: \"05330706-8231-4c38-be56-416f243992c3\") " pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:38.689588 master-1 kubenswrapper[4771]: I1011 10:37:38.689249 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-authentication/oauth-openshift-68fb97bcc4-g7k57"] Oct 11 10:37:38.699787 master-1 kubenswrapper[4771]: I1011 10:37:38.699758 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 10:37:38.749561 master-1 kubenswrapper[4771]: I1011 10:37:38.749499 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp"] Oct 11 10:37:38.759853 master-1 kubenswrapper[4771]: W1011 10:37:38.759568 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd472d171_a5c8_4c71_9d31_7ec0aa3a6db9.slice/crio-afe1b08971e2d05f2fa0425c6f9d72ddbe0592bbee9296deb75b0c968bee6378 WatchSource:0}: Error finding container afe1b08971e2d05f2fa0425c6f9d72ddbe0592bbee9296deb75b0c968bee6378: Status 404 returned error can't find the container with id afe1b08971e2d05f2fa0425c6f9d72ddbe0592bbee9296deb75b0c968bee6378 Oct 11 10:37:38.810170 master-1 kubenswrapper[4771]: I1011 10:37:38.809089 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:39.224746 master-1 kubenswrapper[4771]: I1011 10:37:39.224652 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-6768b5f5f9-r74mm"] Oct 11 10:37:39.227389 master-1 kubenswrapper[4771]: I1011 10:37:39.226434 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-5-master-1" event={"ID":"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4","Type":"ContainerStarted","Data":"5a01f085e3b1c7fb42b9e1bcc547086d47f6a110bdf85f6e451a5f626e8ea9d3"} Oct 11 10:37:39.227389 master-1 kubenswrapper[4771]: I1011 10:37:39.226523 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-5-master-1" event={"ID":"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4","Type":"ContainerStarted","Data":"d51941eb09c3e4a595d1102ccd7dab8966ed8157ff482e09bff14b3b01ba141c"} Oct 11 10:37:39.230148 master-1 kubenswrapper[4771]: I1011 10:37:39.230098 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" event={"ID":"dd18178e-3cb1-41de-8866-913f8f23d90d","Type":"ContainerStarted","Data":"15010edee5601765abeb1b9d8fe25dbc88b3ecac08ba70f44f3a0b0d863ba20f"} Oct 11 10:37:39.230224 master-1 kubenswrapper[4771]: I1011 10:37:39.230159 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" event={"ID":"dd18178e-3cb1-41de-8866-913f8f23d90d","Type":"ContainerStarted","Data":"e3c94e0f6f964ee9f94da14b1bdf57bd8704f7e66d4160744c45e29823a9eaca"} Oct 11 10:37:39.230563 master-1 kubenswrapper[4771]: I1011 10:37:39.230525 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:37:39.232262 master-1 kubenswrapper[4771]: W1011 10:37:39.232220 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05330706_8231_4c38_be56_416f243992c3.slice/crio-61f4f1f6f8e0d63ba26153940ab45c54232f6c30fa1b90435ae48a365377cb35 WatchSource:0}: Error finding container 61f4f1f6f8e0d63ba26153940ab45c54232f6c30fa1b90435ae48a365377cb35: Status 404 returned error can't find the container with id 61f4f1f6f8e0d63ba26153940ab45c54232f6c30fa1b90435ae48a365377cb35 Oct 11 10:37:39.233122 master-1 kubenswrapper[4771]: I1011 10:37:39.233080 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" event={"ID":"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8","Type":"ContainerStarted","Data":"8893531f4f4ebb704c0ef08b506e4add808eee05f66a87ee0d2eb8eddb5d49b0"} Oct 11 10:37:39.235073 master-1 kubenswrapper[4771]: I1011 10:37:39.235020 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" event={"ID":"d472d171-a5c8-4c71-9d31-7ec0aa3a6db9","Type":"ContainerStarted","Data":"afe1b08971e2d05f2fa0425c6f9d72ddbe0592bbee9296deb75b0c968bee6378"} Oct 11 10:37:39.253114 master-1 kubenswrapper[4771]: I1011 10:37:39.253044 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/installer-5-master-1" podStartSLOduration=57.25302427 podStartE2EDuration="57.25302427s" podCreationTimestamp="2025-10-11 10:36:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:37:39.249432178 +0000 UTC m=+691.223658659" watchObservedRunningTime="2025-10-11 10:37:39.25302427 +0000 UTC m=+691.227250711" Oct 11 10:37:39.280689 master-1 kubenswrapper[4771]: I1011 10:37:39.280082 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" podStartSLOduration=162.280058343 podStartE2EDuration="2m42.280058343s" podCreationTimestamp="2025-10-11 10:34:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:37:39.276624175 +0000 UTC m=+691.250850656" watchObservedRunningTime="2025-10-11 10:37:39.280058343 +0000 UTC m=+691.254284804" Oct 11 10:37:40.246904 master-1 kubenswrapper[4771]: I1011 10:37:40.246845 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" event={"ID":"05330706-8231-4c38-be56-416f243992c3","Type":"ContainerStarted","Data":"61f4f1f6f8e0d63ba26153940ab45c54232f6c30fa1b90435ae48a365377cb35"} Oct 11 10:37:42.259131 master-1 kubenswrapper[4771]: I1011 10:37:42.259056 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" event={"ID":"d472d171-a5c8-4c71-9d31-7ec0aa3a6db9","Type":"ContainerStarted","Data":"929913ef2840fcd61f8266001999ff45886c89e4544b76d9a33c10d3318cb69e"} Oct 11 10:37:42.261112 master-1 kubenswrapper[4771]: I1011 10:37:42.261064 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" event={"ID":"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8","Type":"ContainerStarted","Data":"33d94985a7a913ae973d8eae7753333b859741460e8780a5827eed5110a86a93"} Oct 11 10:37:42.261397 master-1 kubenswrapper[4771]: I1011 10:37:42.261347 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:42.265514 master-1 kubenswrapper[4771]: I1011 10:37:42.265472 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:37:42.296296 master-1 kubenswrapper[4771]: I1011 10:37:42.296190 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" podStartSLOduration=2.9128345319999998 podStartE2EDuration="5.296166184s" podCreationTimestamp="2025-10-11 10:37:37 +0000 UTC" firstStartedPulling="2025-10-11 10:37:38.762516588 +0000 UTC m=+690.736743029" lastFinishedPulling="2025-10-11 10:37:41.14584824 +0000 UTC m=+693.120074681" observedRunningTime="2025-10-11 10:37:42.290171493 +0000 UTC m=+694.264397934" watchObservedRunningTime="2025-10-11 10:37:42.296166184 +0000 UTC m=+694.270392625" Oct 11 10:37:42.340927 master-1 kubenswrapper[4771]: I1011 10:37:42.340853 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" podStartSLOduration=7.877255046 podStartE2EDuration="10.340834691s" podCreationTimestamp="2025-10-11 10:37:32 +0000 UTC" firstStartedPulling="2025-10-11 10:37:38.699708093 +0000 UTC m=+690.673934534" lastFinishedPulling="2025-10-11 10:37:41.163287738 +0000 UTC m=+693.137514179" observedRunningTime="2025-10-11 10:37:42.335921 +0000 UTC m=+694.310147461" watchObservedRunningTime="2025-10-11 10:37:42.340834691 +0000 UTC m=+694.315061132" Oct 11 10:37:43.137725 master-1 kubenswrapper[4771]: I1011 10:37:43.137541 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-77c7855cb4-qkp68"] Oct 11 10:37:43.138048 master-1 kubenswrapper[4771]: I1011 10:37:43.137755 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" podUID="e23d9d43-9980-4c16-91c4-9fc0bca161e6" containerName="controller-manager" containerID="cri-o://c570237a7e93abdb8d6cb4489a86eb34cb5e25db0de47a00c9bf05de3a2ba3c4" gracePeriod=30 Oct 11 10:37:43.244587 master-1 kubenswrapper[4771]: I1011 10:37:43.244514 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-authentication/oauth-openshift-68fb97bcc4-g7k57"] Oct 11 10:37:43.271759 master-1 kubenswrapper[4771]: I1011 10:37:43.271694 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" event={"ID":"05330706-8231-4c38-be56-416f243992c3","Type":"ContainerStarted","Data":"6cbe0b22e8526c0b78798635a3f5d8c0f8b018506b067842e391cfdf9bf5d7a5"} Oct 11 10:37:43.272806 master-1 kubenswrapper[4771]: I1011 10:37:43.272574 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:43.281603 master-1 kubenswrapper[4771]: I1011 10:37:43.281546 4771 generic.go:334] "Generic (PLEG): container finished" podID="e23d9d43-9980-4c16-91c4-9fc0bca161e6" containerID="c570237a7e93abdb8d6cb4489a86eb34cb5e25db0de47a00c9bf05de3a2ba3c4" exitCode=0 Oct 11 10:37:43.281754 master-1 kubenswrapper[4771]: I1011 10:37:43.281656 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" event={"ID":"e23d9d43-9980-4c16-91c4-9fc0bca161e6","Type":"ContainerDied","Data":"c570237a7e93abdb8d6cb4489a86eb34cb5e25db0de47a00c9bf05de3a2ba3c4"} Oct 11 10:37:43.283424 master-1 kubenswrapper[4771]: I1011 10:37:43.282978 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" Oct 11 10:37:43.283871 master-1 kubenswrapper[4771]: I1011 10:37:43.283599 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" Oct 11 10:37:43.291150 master-1 kubenswrapper[4771]: I1011 10:37:43.288960 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-578f8b47b8-tljlp" Oct 11 10:37:43.326946 master-1 kubenswrapper[4771]: I1011 10:37:43.321675 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-6768b5f5f9-r74mm" podStartSLOduration=3.054320858 podStartE2EDuration="6.32165564s" podCreationTimestamp="2025-10-11 10:37:37 +0000 UTC" firstStartedPulling="2025-10-11 10:37:39.236042865 +0000 UTC m=+691.210269306" lastFinishedPulling="2025-10-11 10:37:42.503377417 +0000 UTC m=+694.477604088" observedRunningTime="2025-10-11 10:37:43.314039622 +0000 UTC m=+695.288266073" watchObservedRunningTime="2025-10-11 10:37:43.32165564 +0000 UTC m=+695.295882081" Oct 11 10:37:43.342847 master-1 kubenswrapper[4771]: I1011 10:37:43.342741 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m"] Oct 11 10:37:43.343286 master-1 kubenswrapper[4771]: I1011 10:37:43.343199 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" podUID="e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" containerName="route-controller-manager" containerID="cri-o://5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547" gracePeriod=30 Oct 11 10:37:43.557190 master-1 kubenswrapper[4771]: I1011 10:37:43.555645 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-65bb9777fc-66jxg"] Oct 11 10:37:43.557190 master-1 kubenswrapper[4771]: I1011 10:37:43.556302 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-65bb9777fc-66jxg" Oct 11 10:37:43.558331 master-1 kubenswrapper[4771]: I1011 10:37:43.558258 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"openshift-service-ca.crt" Oct 11 10:37:43.558493 master-1 kubenswrapper[4771]: I1011 10:37:43.558472 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"default-dockercfg-pqtx8" Oct 11 10:37:43.560005 master-1 kubenswrapper[4771]: I1011 10:37:43.559259 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"kube-root-ca.crt" Oct 11 10:37:43.560088 master-1 kubenswrapper[4771]: I1011 10:37:43.560047 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:37:43.586133 master-1 kubenswrapper[4771]: I1011 10:37:43.586077 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-65bb9777fc-66jxg"] Oct 11 10:37:43.688132 master-1 kubenswrapper[4771]: I1011 10:37:43.688081 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e23d9d43-9980-4c16-91c4-9fc0bca161e6-serving-cert\") pod \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " Oct 11 10:37:43.688132 master-1 kubenswrapper[4771]: I1011 10:37:43.688130 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-proxy-ca-bundles\") pod \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " Oct 11 10:37:43.688378 master-1 kubenswrapper[4771]: I1011 10:37:43.688169 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-client-ca\") pod \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " Oct 11 10:37:43.688378 master-1 kubenswrapper[4771]: I1011 10:37:43.688192 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwlwv\" (UniqueName: \"kubernetes.io/projected/e23d9d43-9980-4c16-91c4-9fc0bca161e6-kube-api-access-fwlwv\") pod \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " Oct 11 10:37:43.688378 master-1 kubenswrapper[4771]: I1011 10:37:43.688238 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-config\") pod \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\" (UID: \"e23d9d43-9980-4c16-91c4-9fc0bca161e6\") " Oct 11 10:37:43.688470 master-1 kubenswrapper[4771]: I1011 10:37:43.688412 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqpr\" (UniqueName: \"kubernetes.io/projected/6958daf7-e9a7-4151-8e42-851feedec58e-kube-api-access-5mqpr\") pod \"downloads-65bb9777fc-66jxg\" (UID: \"6958daf7-e9a7-4151-8e42-851feedec58e\") " pod="openshift-console/downloads-65bb9777fc-66jxg" Oct 11 10:37:43.689753 master-1 kubenswrapper[4771]: I1011 10:37:43.689488 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-client-ca" (OuterVolumeSpecName: "client-ca") pod "e23d9d43-9980-4c16-91c4-9fc0bca161e6" (UID: "e23d9d43-9980-4c16-91c4-9fc0bca161e6"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:37:43.690206 master-1 kubenswrapper[4771]: I1011 10:37:43.690175 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-proxy-ca-bundles" (OuterVolumeSpecName: "proxy-ca-bundles") pod "e23d9d43-9980-4c16-91c4-9fc0bca161e6" (UID: "e23d9d43-9980-4c16-91c4-9fc0bca161e6"). InnerVolumeSpecName "proxy-ca-bundles". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:37:43.691494 master-1 kubenswrapper[4771]: I1011 10:37:43.691142 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-config" (OuterVolumeSpecName: "config") pod "e23d9d43-9980-4c16-91c4-9fc0bca161e6" (UID: "e23d9d43-9980-4c16-91c4-9fc0bca161e6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:37:43.691494 master-1 kubenswrapper[4771]: I1011 10:37:43.691283 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23d9d43-9980-4c16-91c4-9fc0bca161e6-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "e23d9d43-9980-4c16-91c4-9fc0bca161e6" (UID: "e23d9d43-9980-4c16-91c4-9fc0bca161e6"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:37:43.694517 master-1 kubenswrapper[4771]: I1011 10:37:43.694491 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23d9d43-9980-4c16-91c4-9fc0bca161e6-kube-api-access-fwlwv" (OuterVolumeSpecName: "kube-api-access-fwlwv") pod "e23d9d43-9980-4c16-91c4-9fc0bca161e6" (UID: "e23d9d43-9980-4c16-91c4-9fc0bca161e6"). InnerVolumeSpecName "kube-api-access-fwlwv". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:37:43.791061 master-1 kubenswrapper[4771]: I1011 10:37:43.789392 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqpr\" (UniqueName: \"kubernetes.io/projected/6958daf7-e9a7-4151-8e42-851feedec58e-kube-api-access-5mqpr\") pod \"downloads-65bb9777fc-66jxg\" (UID: \"6958daf7-e9a7-4151-8e42-851feedec58e\") " pod="openshift-console/downloads-65bb9777fc-66jxg" Oct 11 10:37:43.791061 master-1 kubenswrapper[4771]: I1011 10:37:43.790021 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:43.791061 master-1 kubenswrapper[4771]: I1011 10:37:43.790035 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e23d9d43-9980-4c16-91c4-9fc0bca161e6-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:43.791061 master-1 kubenswrapper[4771]: I1011 10:37:43.790049 4771 reconciler_common.go:293] "Volume detached for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-proxy-ca-bundles\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:43.791061 master-1 kubenswrapper[4771]: I1011 10:37:43.790058 4771 reconciler_common.go:293] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e23d9d43-9980-4c16-91c4-9fc0bca161e6-client-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:43.791061 master-1 kubenswrapper[4771]: I1011 10:37:43.790067 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-fwlwv\" (UniqueName: \"kubernetes.io/projected/e23d9d43-9980-4c16-91c4-9fc0bca161e6-kube-api-access-fwlwv\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:43.823594 master-1 kubenswrapper[4771]: I1011 10:37:43.823534 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:37:43.833651 master-1 kubenswrapper[4771]: I1011 10:37:43.833580 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqpr\" (UniqueName: \"kubernetes.io/projected/6958daf7-e9a7-4151-8e42-851feedec58e-kube-api-access-5mqpr\") pod \"downloads-65bb9777fc-66jxg\" (UID: \"6958daf7-e9a7-4151-8e42-851feedec58e\") " pod="openshift-console/downloads-65bb9777fc-66jxg" Oct 11 10:37:43.871079 master-1 kubenswrapper[4771]: I1011 10:37:43.871025 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-65bb9777fc-66jxg" Oct 11 10:37:43.999454 master-1 kubenswrapper[4771]: I1011 10:37:43.994807 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-serving-cert\") pod \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " Oct 11 10:37:43.999454 master-1 kubenswrapper[4771]: I1011 10:37:43.994853 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zmdm\" (UniqueName: \"kubernetes.io/projected/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-kube-api-access-9zmdm\") pod \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " Oct 11 10:37:43.999454 master-1 kubenswrapper[4771]: I1011 10:37:43.994883 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-config\") pod \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " Oct 11 10:37:43.999454 master-1 kubenswrapper[4771]: I1011 10:37:43.994929 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-client-ca\") pod \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\" (UID: \"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3\") " Oct 11 10:37:43.999454 master-1 kubenswrapper[4771]: I1011 10:37:43.995610 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-client-ca" (OuterVolumeSpecName: "client-ca") pod "e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" (UID: "e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:37:43.999454 master-1 kubenswrapper[4771]: I1011 10:37:43.995739 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-config" (OuterVolumeSpecName: "config") pod "e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" (UID: "e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:37:43.999454 master-1 kubenswrapper[4771]: I1011 10:37:43.997854 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" (UID: "e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:37:43.999454 master-1 kubenswrapper[4771]: I1011 10:37:43.998699 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-kube-api-access-9zmdm" (OuterVolumeSpecName: "kube-api-access-9zmdm") pod "e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" (UID: "e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3"). InnerVolumeSpecName "kube-api-access-9zmdm". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:37:44.096439 master-1 kubenswrapper[4771]: I1011 10:37:44.096381 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:44.096439 master-1 kubenswrapper[4771]: I1011 10:37:44.096430 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:44.096439 master-1 kubenswrapper[4771]: I1011 10:37:44.096443 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-9zmdm\" (UniqueName: \"kubernetes.io/projected/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-kube-api-access-9zmdm\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:44.096439 master-1 kubenswrapper[4771]: I1011 10:37:44.096455 4771 reconciler_common.go:293] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3-client-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:37:44.289772 master-1 kubenswrapper[4771]: I1011 10:37:44.289630 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" Oct 11 10:37:44.293558 master-1 kubenswrapper[4771]: I1011 10:37:44.293492 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-77c7855cb4-qkp68" event={"ID":"e23d9d43-9980-4c16-91c4-9fc0bca161e6","Type":"ContainerDied","Data":"5577977e3fcec143fb9fe4819b109c252e41520252e8f2be4cdb67371fc4b2fd"} Oct 11 10:37:44.293684 master-1 kubenswrapper[4771]: I1011 10:37:44.293573 4771 scope.go:117] "RemoveContainer" containerID="c570237a7e93abdb8d6cb4489a86eb34cb5e25db0de47a00c9bf05de3a2ba3c4" Oct 11 10:37:44.295889 master-1 kubenswrapper[4771]: I1011 10:37:44.295830 4771 generic.go:334] "Generic (PLEG): container finished" podID="e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" containerID="5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547" exitCode=0 Oct 11 10:37:44.296601 master-1 kubenswrapper[4771]: I1011 10:37:44.296567 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" Oct 11 10:37:44.298600 master-1 kubenswrapper[4771]: I1011 10:37:44.298534 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" event={"ID":"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3","Type":"ContainerDied","Data":"5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547"} Oct 11 10:37:44.298600 master-1 kubenswrapper[4771]: I1011 10:37:44.298582 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m" event={"ID":"e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3","Type":"ContainerDied","Data":"fb8e606f605b5a7a5119eb59ac6c30ff451c4fbab3f45cf0454534a92053916c"} Oct 11 10:37:44.310669 master-1 kubenswrapper[4771]: I1011 10:37:44.310619 4771 scope.go:117] "RemoveContainer" containerID="5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547" Oct 11 10:37:44.323657 master-1 kubenswrapper[4771]: I1011 10:37:44.323617 4771 scope.go:117] "RemoveContainer" containerID="5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547" Oct 11 10:37:44.324057 master-1 kubenswrapper[4771]: E1011 10:37:44.324019 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547\": container with ID starting with 5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547 not found: ID does not exist" containerID="5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547" Oct 11 10:37:44.324109 master-1 kubenswrapper[4771]: I1011 10:37:44.324053 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547"} err="failed to get container status \"5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547\": rpc error: code = NotFound desc = could not find container \"5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547\": container with ID starting with 5088293846da37543ad3781b1214351a957d3cdeb729f2d5b67c96a0a56aa547 not found: ID does not exist" Oct 11 10:37:44.369486 master-1 kubenswrapper[4771]: I1011 10:37:44.369423 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-65bb9777fc-66jxg"] Oct 11 10:37:44.382161 master-1 kubenswrapper[4771]: W1011 10:37:44.382099 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6958daf7_e9a7_4151_8e42_851feedec58e.slice/crio-e2a80dd3d13e98b0d9cb34b6b2710b9cd6757b9002e8ad477d086d18cf635f54 WatchSource:0}: Error finding container e2a80dd3d13e98b0d9cb34b6b2710b9cd6757b9002e8ad477d086d18cf635f54: Status 404 returned error can't find the container with id e2a80dd3d13e98b0d9cb34b6b2710b9cd6757b9002e8ad477d086d18cf635f54 Oct 11 10:37:44.409394 master-1 kubenswrapper[4771]: I1011 10:37:44.409301 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-controller-manager/controller-manager-77c7855cb4-qkp68"] Oct 11 10:37:44.422192 master-1 kubenswrapper[4771]: I1011 10:37:44.422125 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-controller-manager/controller-manager-77c7855cb4-qkp68"] Oct 11 10:37:44.450613 master-1 kubenswrapper[4771]: I1011 10:37:44.450529 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23d9d43-9980-4c16-91c4-9fc0bca161e6" path="/var/lib/kubelet/pods/e23d9d43-9980-4c16-91c4-9fc0bca161e6/volumes" Oct 11 10:37:44.459784 master-1 kubenswrapper[4771]: I1011 10:37:44.459703 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m"] Oct 11 10:37:44.475951 master-1 kubenswrapper[4771]: I1011 10:37:44.475891 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-68b68f45cd-mqn2m"] Oct 11 10:37:44.668253 master-1 kubenswrapper[4771]: I1011 10:37:44.668184 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-controller-manager/controller-manager-897b595f-6mkbk"] Oct 11 10:37:44.668599 master-1 kubenswrapper[4771]: E1011 10:37:44.668461 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" containerName="route-controller-manager" Oct 11 10:37:44.668599 master-1 kubenswrapper[4771]: I1011 10:37:44.668481 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" containerName="route-controller-manager" Oct 11 10:37:44.668599 master-1 kubenswrapper[4771]: E1011 10:37:44.668494 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e23d9d43-9980-4c16-91c4-9fc0bca161e6" containerName="controller-manager" Oct 11 10:37:44.668599 master-1 kubenswrapper[4771]: I1011 10:37:44.668504 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23d9d43-9980-4c16-91c4-9fc0bca161e6" containerName="controller-manager" Oct 11 10:37:44.668950 master-1 kubenswrapper[4771]: I1011 10:37:44.668626 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" containerName="route-controller-manager" Oct 11 10:37:44.668950 master-1 kubenswrapper[4771]: I1011 10:37:44.668646 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e23d9d43-9980-4c16-91c4-9fc0bca161e6" containerName="controller-manager" Oct 11 10:37:44.671537 master-1 kubenswrapper[4771]: I1011 10:37:44.669216 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.671537 master-1 kubenswrapper[4771]: I1011 10:37:44.671402 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"client-ca" Oct 11 10:37:44.671735 master-1 kubenswrapper[4771]: I1011 10:37:44.671678 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-controller-manager"/"serving-cert" Oct 11 10:37:44.672806 master-1 kubenswrapper[4771]: I1011 10:37:44.672218 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"kube-root-ca.crt" Oct 11 10:37:44.672806 master-1 kubenswrapper[4771]: I1011 10:37:44.672417 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"config" Oct 11 10:37:44.673986 master-1 kubenswrapper[4771]: I1011 10:37:44.673494 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-service-ca.crt" Oct 11 10:37:44.676773 master-1 kubenswrapper[4771]: I1011 10:37:44.676709 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-controller-manager"/"openshift-controller-manager-sa-dockercfg-z46lz" Oct 11 10:37:44.683667 master-1 kubenswrapper[4771]: I1011 10:37:44.682990 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29"] Oct 11 10:37:44.683926 master-1 kubenswrapper[4771]: I1011 10:37:44.683855 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.683978 master-1 kubenswrapper[4771]: I1011 10:37:44.683923 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-controller-manager"/"openshift-global-ca" Oct 11 10:37:44.724239 master-1 kubenswrapper[4771]: I1011 10:37:44.686711 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"client-ca" Oct 11 10:37:44.724239 master-1 kubenswrapper[4771]: I1011 10:37:44.686919 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-route-controller-manager"/"serving-cert" Oct 11 10:37:44.724239 master-1 kubenswrapper[4771]: I1011 10:37:44.687466 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"config" Oct 11 10:37:44.724239 master-1 kubenswrapper[4771]: I1011 10:37:44.687678 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"kube-root-ca.crt" Oct 11 10:37:44.724239 master-1 kubenswrapper[4771]: I1011 10:37:44.687836 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-route-controller-manager"/"route-controller-manager-sa-dockercfg-vwjkz" Oct 11 10:37:44.724239 master-1 kubenswrapper[4771]: I1011 10:37:44.688018 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-route-controller-manager"/"openshift-service-ca.crt" Oct 11 10:37:44.724239 master-1 kubenswrapper[4771]: I1011 10:37:44.690929 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-controller-manager/controller-manager-897b595f-6mkbk"] Oct 11 10:37:44.724239 master-1 kubenswrapper[4771]: I1011 10:37:44.706049 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29"] Oct 11 10:37:44.827306 master-1 kubenswrapper[4771]: I1011 10:37:44.827165 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kn6\" (UniqueName: \"kubernetes.io/projected/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-kube-api-access-k5kn6\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.827517 master-1 kubenswrapper[4771]: I1011 10:37:44.827345 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-config\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.827517 master-1 kubenswrapper[4771]: I1011 10:37:44.827404 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-serving-cert\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.827517 master-1 kubenswrapper[4771]: I1011 10:37:44.827432 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dfa56da-3482-4730-ac7a-311905e7396d-config\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.827517 master-1 kubenswrapper[4771]: I1011 10:37:44.827495 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66tpk\" (UniqueName: \"kubernetes.io/projected/7dfa56da-3482-4730-ac7a-311905e7396d-kube-api-access-66tpk\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.827766 master-1 kubenswrapper[4771]: I1011 10:37:44.827552 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-client-ca\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.827766 master-1 kubenswrapper[4771]: I1011 10:37:44.827579 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-proxy-ca-bundles\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.827766 master-1 kubenswrapper[4771]: I1011 10:37:44.827601 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa56da-3482-4730-ac7a-311905e7396d-client-ca\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.827766 master-1 kubenswrapper[4771]: I1011 10:37:44.827706 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dfa56da-3482-4730-ac7a-311905e7396d-serving-cert\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.928720 master-1 kubenswrapper[4771]: I1011 10:37:44.928655 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dfa56da-3482-4730-ac7a-311905e7396d-serving-cert\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.928720 master-1 kubenswrapper[4771]: I1011 10:37:44.928722 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kn6\" (UniqueName: \"kubernetes.io/projected/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-kube-api-access-k5kn6\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.928970 master-1 kubenswrapper[4771]: I1011 10:37:44.928744 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-config\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.928970 master-1 kubenswrapper[4771]: I1011 10:37:44.928760 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-serving-cert\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.928970 master-1 kubenswrapper[4771]: I1011 10:37:44.928782 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dfa56da-3482-4730-ac7a-311905e7396d-config\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.928970 master-1 kubenswrapper[4771]: I1011 10:37:44.928806 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-66tpk\" (UniqueName: \"kubernetes.io/projected/7dfa56da-3482-4730-ac7a-311905e7396d-kube-api-access-66tpk\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.928970 master-1 kubenswrapper[4771]: I1011 10:37:44.928822 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-client-ca\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.928970 master-1 kubenswrapper[4771]: I1011 10:37:44.928837 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-proxy-ca-bundles\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.928970 master-1 kubenswrapper[4771]: I1011 10:37:44.928855 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa56da-3482-4730-ac7a-311905e7396d-client-ca\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.930481 master-1 kubenswrapper[4771]: I1011 10:37:44.930444 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa56da-3482-4730-ac7a-311905e7396d-client-ca\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.930610 master-1 kubenswrapper[4771]: I1011 10:37:44.930588 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-client-ca\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.930718 master-1 kubenswrapper[4771]: I1011 10:37:44.930652 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dfa56da-3482-4730-ac7a-311905e7396d-config\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.931155 master-1 kubenswrapper[4771]: I1011 10:37:44.930977 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"proxy-ca-bundles\" (UniqueName: \"kubernetes.io/configmap/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-proxy-ca-bundles\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.932060 master-1 kubenswrapper[4771]: I1011 10:37:44.931974 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-config\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.932641 master-1 kubenswrapper[4771]: I1011 10:37:44.932582 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dfa56da-3482-4730-ac7a-311905e7396d-serving-cert\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.932792 master-1 kubenswrapper[4771]: I1011 10:37:44.932751 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-serving-cert\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.954094 master-1 kubenswrapper[4771]: I1011 10:37:44.954023 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-66tpk\" (UniqueName: \"kubernetes.io/projected/7dfa56da-3482-4730-ac7a-311905e7396d-kube-api-access-66tpk\") pod \"route-controller-manager-57c8488cd7-5ld29\" (UID: \"7dfa56da-3482-4730-ac7a-311905e7396d\") " pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:44.962318 master-1 kubenswrapper[4771]: I1011 10:37:44.962247 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kn6\" (UniqueName: \"kubernetes.io/projected/9524eebe-db33-4399-8fe5-0bcfd9fbd9f4-kube-api-access-k5kn6\") pod \"controller-manager-897b595f-6mkbk\" (UID: \"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4\") " pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:44.985229 master-1 kubenswrapper[4771]: I1011 10:37:44.985163 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:45.036764 master-1 kubenswrapper[4771]: I1011 10:37:45.036691 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:45.307461 master-1 kubenswrapper[4771]: I1011 10:37:45.307389 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-65bb9777fc-66jxg" event={"ID":"6958daf7-e9a7-4151-8e42-851feedec58e","Type":"ContainerStarted","Data":"e2a80dd3d13e98b0d9cb34b6b2710b9cd6757b9002e8ad477d086d18cf635f54"} Oct 11 10:37:45.427202 master-1 kubenswrapper[4771]: I1011 10:37:45.427129 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-controller-manager/controller-manager-897b595f-6mkbk"] Oct 11 10:37:45.431044 master-1 kubenswrapper[4771]: W1011 10:37:45.430985 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9524eebe_db33_4399_8fe5_0bcfd9fbd9f4.slice/crio-1e236066b0bc7b8da43a96fd9fcbddda3e1776a867545aa62fb4bcb21bbe3f38 WatchSource:0}: Error finding container 1e236066b0bc7b8da43a96fd9fcbddda3e1776a867545aa62fb4bcb21bbe3f38: Status 404 returned error can't find the container with id 1e236066b0bc7b8da43a96fd9fcbddda3e1776a867545aa62fb4bcb21bbe3f38 Oct 11 10:37:45.513902 master-1 kubenswrapper[4771]: I1011 10:37:45.513810 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29"] Oct 11 10:37:45.524994 master-1 kubenswrapper[4771]: W1011 10:37:45.524927 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfa56da_3482_4730_ac7a_311905e7396d.slice/crio-402b1304f304577e40efaeb530aa7b425f38dc76d456006b9d07ff74bd80c578 WatchSource:0}: Error finding container 402b1304f304577e40efaeb530aa7b425f38dc76d456006b9d07ff74bd80c578: Status 404 returned error can't find the container with id 402b1304f304577e40efaeb530aa7b425f38dc76d456006b9d07ff74bd80c578 Oct 11 10:37:46.322214 master-1 kubenswrapper[4771]: I1011 10:37:46.322141 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" event={"ID":"7dfa56da-3482-4730-ac7a-311905e7396d","Type":"ContainerStarted","Data":"ad938e460e23dec7fb0a3338327dbffbabede133f5cc566ab9352d8427863862"} Oct 11 10:37:46.322765 master-1 kubenswrapper[4771]: I1011 10:37:46.322290 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" event={"ID":"7dfa56da-3482-4730-ac7a-311905e7396d","Type":"ContainerStarted","Data":"402b1304f304577e40efaeb530aa7b425f38dc76d456006b9d07ff74bd80c578"} Oct 11 10:37:46.322765 master-1 kubenswrapper[4771]: I1011 10:37:46.322335 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:46.324957 master-1 kubenswrapper[4771]: I1011 10:37:46.324912 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" event={"ID":"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4","Type":"ContainerStarted","Data":"a442af9c775f277be6982e731bddbf355cf0e3158bf4e9c3f9821d19f824caa6"} Oct 11 10:37:46.325052 master-1 kubenswrapper[4771]: I1011 10:37:46.324968 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" event={"ID":"9524eebe-db33-4399-8fe5-0bcfd9fbd9f4","Type":"ContainerStarted","Data":"1e236066b0bc7b8da43a96fd9fcbddda3e1776a867545aa62fb4bcb21bbe3f38"} Oct 11 10:37:46.325107 master-1 kubenswrapper[4771]: I1011 10:37:46.325093 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:46.328927 master-1 kubenswrapper[4771]: I1011 10:37:46.328888 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" Oct 11 10:37:46.332094 master-1 kubenswrapper[4771]: I1011 10:37:46.332053 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" Oct 11 10:37:46.387745 master-1 kubenswrapper[4771]: I1011 10:37:46.387665 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-route-controller-manager/route-controller-manager-57c8488cd7-5ld29" podStartSLOduration=3.387648597 podStartE2EDuration="3.387648597s" podCreationTimestamp="2025-10-11 10:37:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:37:46.38567317 +0000 UTC m=+698.359899611" watchObservedRunningTime="2025-10-11 10:37:46.387648597 +0000 UTC m=+698.361875038" Oct 11 10:37:46.442931 master-1 kubenswrapper[4771]: I1011 10:37:46.442883 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3" path="/var/lib/kubelet/pods/e89d5fa2-4b2d-47b8-9f43-fbf5942eaff3/volumes" Oct 11 10:37:46.456338 master-1 kubenswrapper[4771]: I1011 10:37:46.456110 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-controller-manager/controller-manager-897b595f-6mkbk" podStartSLOduration=3.456092693 podStartE2EDuration="3.456092693s" podCreationTimestamp="2025-10-11 10:37:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:37:46.4552885 +0000 UTC m=+698.429514971" watchObservedRunningTime="2025-10-11 10:37:46.456092693 +0000 UTC m=+698.430319134" Oct 11 10:37:46.833398 master-1 kubenswrapper[4771]: I1011 10:37:46.833313 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-console/console-57bccbfdf6-l962w"] Oct 11 10:37:46.834120 master-1 kubenswrapper[4771]: I1011 10:37:46.834089 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.838259 master-1 kubenswrapper[4771]: I1011 10:37:46.838231 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"console-oauth-config" Oct 11 10:37:46.838663 master-1 kubenswrapper[4771]: I1011 10:37:46.838643 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"console-serving-cert" Oct 11 10:37:46.838853 master-1 kubenswrapper[4771]: I1011 10:37:46.838790 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"oauth-serving-cert" Oct 11 10:37:46.840378 master-1 kubenswrapper[4771]: I1011 10:37:46.840338 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"service-ca" Oct 11 10:37:46.840678 master-1 kubenswrapper[4771]: I1011 10:37:46.840644 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"console-dockercfg-8wmjp" Oct 11 10:37:46.842234 master-1 kubenswrapper[4771]: I1011 10:37:46.842204 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"console-config" Oct 11 10:37:46.858888 master-1 kubenswrapper[4771]: I1011 10:37:46.858811 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-console-config\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.859091 master-1 kubenswrapper[4771]: I1011 10:37:46.858935 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-oauth-config\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.859091 master-1 kubenswrapper[4771]: I1011 10:37:46.858998 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-serving-cert\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.859091 master-1 kubenswrapper[4771]: I1011 10:37:46.859031 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-oauth-serving-cert\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.859241 master-1 kubenswrapper[4771]: I1011 10:37:46.859105 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngk7c\" (UniqueName: \"kubernetes.io/projected/a7c05954-4353-4cd1-9130-7fcb832a0493-kube-api-access-ngk7c\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.859241 master-1 kubenswrapper[4771]: I1011 10:37:46.859184 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-service-ca\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.866060 master-1 kubenswrapper[4771]: I1011 10:37:46.866004 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57bccbfdf6-l962w"] Oct 11 10:37:46.960591 master-1 kubenswrapper[4771]: I1011 10:37:46.960543 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-console-config\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.960913 master-1 kubenswrapper[4771]: I1011 10:37:46.960895 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-oauth-config\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.961012 master-1 kubenswrapper[4771]: I1011 10:37:46.960996 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-serving-cert\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.961147 master-1 kubenswrapper[4771]: I1011 10:37:46.961131 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-oauth-serving-cert\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.961273 master-1 kubenswrapper[4771]: I1011 10:37:46.961257 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk7c\" (UniqueName: \"kubernetes.io/projected/a7c05954-4353-4cd1-9130-7fcb832a0493-kube-api-access-ngk7c\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.961419 master-1 kubenswrapper[4771]: I1011 10:37:46.961402 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-service-ca\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.961851 master-1 kubenswrapper[4771]: I1011 10:37:46.961823 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-console-config\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.962194 master-1 kubenswrapper[4771]: I1011 10:37:46.962179 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-service-ca\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.962527 master-1 kubenswrapper[4771]: I1011 10:37:46.962447 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-oauth-serving-cert\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.967524 master-1 kubenswrapper[4771]: I1011 10:37:46.967491 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-serving-cert\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:46.967766 master-1 kubenswrapper[4771]: I1011 10:37:46.967734 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-oauth-config\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:47.038087 master-1 kubenswrapper[4771]: I1011 10:37:47.038037 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngk7c\" (UniqueName: \"kubernetes.io/projected/a7c05954-4353-4cd1-9130-7fcb832a0493-kube-api-access-ngk7c\") pod \"console-57bccbfdf6-l962w\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:47.149730 master-1 kubenswrapper[4771]: I1011 10:37:47.149619 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:37:47.583012 master-1 kubenswrapper[4771]: I1011 10:37:47.582961 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57bccbfdf6-l962w"] Oct 11 10:37:48.338242 master-1 kubenswrapper[4771]: I1011 10:37:48.338150 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bccbfdf6-l962w" event={"ID":"a7c05954-4353-4cd1-9130-7fcb832a0493","Type":"ContainerStarted","Data":"8b80ae3136f1889aef88e388582b0fe2c7b18eb0654ca449f848a798f30b4031"} Oct 11 10:37:49.947455 master-1 kubenswrapper[4771]: I1011 10:37:49.947389 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57bccbfdf6-l962w"] Oct 11 10:37:53.369947 master-1 kubenswrapper[4771]: I1011 10:37:53.369833 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bccbfdf6-l962w" event={"ID":"a7c05954-4353-4cd1-9130-7fcb832a0493","Type":"ContainerStarted","Data":"ac4cbca778491aecc2f71b3dc29feeec7e7dede29f2bc161a327a52e374f391b"} Oct 11 10:37:54.230324 master-1 kubenswrapper[4771]: I1011 10:37:54.230181 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-57bccbfdf6-l962w" podStartSLOduration=3.5835673679999998 podStartE2EDuration="8.23014872s" podCreationTimestamp="2025-10-11 10:37:46 +0000 UTC" firstStartedPulling="2025-10-11 10:37:47.598769498 +0000 UTC m=+699.572995959" lastFinishedPulling="2025-10-11 10:37:52.24535083 +0000 UTC m=+704.219577311" observedRunningTime="2025-10-11 10:37:54.223285514 +0000 UTC m=+706.197511985" watchObservedRunningTime="2025-10-11 10:37:54.23014872 +0000 UTC m=+706.204375191" Oct 11 10:37:57.150839 master-1 kubenswrapper[4771]: I1011 10:37:57.150776 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:38:00.880301 master-1 kubenswrapper[4771]: I1011 10:38:00.880184 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-85df6bdd68-48crk"] Oct 11 10:38:00.881709 master-1 kubenswrapper[4771]: I1011 10:38:00.881654 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" Oct 11 10:38:00.885425 master-1 kubenswrapper[4771]: I1011 10:38:00.885332 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-network-console"/"networking-console-plugin" Oct 11 10:38:00.885683 master-1 kubenswrapper[4771]: I1011 10:38:00.885344 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-network-console"/"networking-console-plugin-cert" Oct 11 10:38:00.894208 master-1 kubenswrapper[4771]: I1011 10:38:00.894158 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-85df6bdd68-48crk"] Oct 11 10:38:00.947847 master-1 kubenswrapper[4771]: I1011 10:38:00.947777 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45c6c687-55ce-4176-903e-5dadd7371470-networking-console-plugin-cert\") pod \"networking-console-plugin-85df6bdd68-48crk\" (UID: \"45c6c687-55ce-4176-903e-5dadd7371470\") " pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" Oct 11 10:38:00.947847 master-1 kubenswrapper[4771]: I1011 10:38:00.947836 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/45c6c687-55ce-4176-903e-5dadd7371470-nginx-conf\") pod \"networking-console-plugin-85df6bdd68-48crk\" (UID: \"45c6c687-55ce-4176-903e-5dadd7371470\") " pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" Oct 11 10:38:01.049185 master-1 kubenswrapper[4771]: I1011 10:38:01.049112 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45c6c687-55ce-4176-903e-5dadd7371470-networking-console-plugin-cert\") pod \"networking-console-plugin-85df6bdd68-48crk\" (UID: \"45c6c687-55ce-4176-903e-5dadd7371470\") " pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" Oct 11 10:38:01.049483 master-1 kubenswrapper[4771]: I1011 10:38:01.049236 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/45c6c687-55ce-4176-903e-5dadd7371470-nginx-conf\") pod \"networking-console-plugin-85df6bdd68-48crk\" (UID: \"45c6c687-55ce-4176-903e-5dadd7371470\") " pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" Oct 11 10:38:01.049483 master-1 kubenswrapper[4771]: E1011 10:38:01.049405 4771 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 11 10:38:01.049613 master-1 kubenswrapper[4771]: E1011 10:38:01.049523 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c6c687-55ce-4176-903e-5dadd7371470-networking-console-plugin-cert podName:45c6c687-55ce-4176-903e-5dadd7371470 nodeName:}" failed. No retries permitted until 2025-10-11 10:38:01.549494293 +0000 UTC m=+713.523720774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/45c6c687-55ce-4176-903e-5dadd7371470-networking-console-plugin-cert") pod "networking-console-plugin-85df6bdd68-48crk" (UID: "45c6c687-55ce-4176-903e-5dadd7371470") : secret "networking-console-plugin-cert" not found Oct 11 10:38:01.050951 master-1 kubenswrapper[4771]: I1011 10:38:01.050885 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/45c6c687-55ce-4176-903e-5dadd7371470-nginx-conf\") pod \"networking-console-plugin-85df6bdd68-48crk\" (UID: \"45c6c687-55ce-4176-903e-5dadd7371470\") " pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" Oct 11 10:38:01.555577 master-1 kubenswrapper[4771]: I1011 10:38:01.555501 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45c6c687-55ce-4176-903e-5dadd7371470-networking-console-plugin-cert\") pod \"networking-console-plugin-85df6bdd68-48crk\" (UID: \"45c6c687-55ce-4176-903e-5dadd7371470\") " pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" Oct 11 10:38:01.559991 master-1 kubenswrapper[4771]: I1011 10:38:01.559936 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45c6c687-55ce-4176-903e-5dadd7371470-networking-console-plugin-cert\") pod \"networking-console-plugin-85df6bdd68-48crk\" (UID: \"45c6c687-55ce-4176-903e-5dadd7371470\") " pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" Oct 11 10:38:01.851103 master-1 kubenswrapper[4771]: I1011 10:38:01.850922 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" Oct 11 10:38:02.330970 master-1 kubenswrapper[4771]: I1011 10:38:02.329432 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-85df6bdd68-48crk"] Oct 11 10:38:06.509824 master-1 kubenswrapper[4771]: I1011 10:38:06.509767 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-1" Oct 11 10:38:09.316783 master-1 kubenswrapper[4771]: I1011 10:38:09.316668 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" podUID="d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" containerName="oauth-openshift" containerID="cri-o://33d94985a7a913ae973d8eae7753333b859741460e8780a5827eed5110a86a93" gracePeriod=15 Oct 11 10:38:09.506395 master-1 kubenswrapper[4771]: I1011 10:38:09.506264 4771 generic.go:334] "Generic (PLEG): container finished" podID="d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" containerID="33d94985a7a913ae973d8eae7753333b859741460e8780a5827eed5110a86a93" exitCode=0 Oct 11 10:38:09.506395 master-1 kubenswrapper[4771]: I1011 10:38:09.506333 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" event={"ID":"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8","Type":"ContainerDied","Data":"33d94985a7a913ae973d8eae7753333b859741460e8780a5827eed5110a86a93"} Oct 11 10:38:11.641253 master-1 kubenswrapper[4771]: I1011 10:38:11.641186 4771 kubelet.go:2431] "SyncLoop REMOVE" source="file" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:38:11.641768 master-1 kubenswrapper[4771]: I1011 10:38:11.641578 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager" containerID="cri-o://bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db" gracePeriod=30 Oct 11 10:38:11.641768 master-1 kubenswrapper[4771]: I1011 10:38:11.641599 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager-recovery-controller" containerID="cri-o://06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1" gracePeriod=30 Oct 11 10:38:11.641768 master-1 kubenswrapper[4771]: I1011 10:38:11.641712 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="cluster-policy-controller" containerID="cri-o://c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb" gracePeriod=30 Oct 11 10:38:11.641768 master-1 kubenswrapper[4771]: I1011 10:38:11.641693 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager-cert-syncer" containerID="cri-o://38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5" gracePeriod=30 Oct 11 10:38:11.642856 master-1 kubenswrapper[4771]: I1011 10:38:11.642684 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:38:11.643048 master-1 kubenswrapper[4771]: E1011 10:38:11.643014 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="cluster-policy-controller" Oct 11 10:38:11.643048 master-1 kubenswrapper[4771]: I1011 10:38:11.643041 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="cluster-policy-controller" Oct 11 10:38:11.643141 master-1 kubenswrapper[4771]: E1011 10:38:11.643066 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager" Oct 11 10:38:11.643141 master-1 kubenswrapper[4771]: I1011 10:38:11.643076 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager" Oct 11 10:38:11.643141 master-1 kubenswrapper[4771]: E1011 10:38:11.643095 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager-cert-syncer" Oct 11 10:38:11.643141 master-1 kubenswrapper[4771]: I1011 10:38:11.643108 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager-cert-syncer" Oct 11 10:38:11.643141 master-1 kubenswrapper[4771]: E1011 10:38:11.643119 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager-recovery-controller" Oct 11 10:38:11.643141 master-1 kubenswrapper[4771]: I1011 10:38:11.643128 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager-recovery-controller" Oct 11 10:38:11.643330 master-1 kubenswrapper[4771]: I1011 10:38:11.643252 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager-recovery-controller" Oct 11 10:38:11.643330 master-1 kubenswrapper[4771]: I1011 10:38:11.643269 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager" Oct 11 10:38:11.643330 master-1 kubenswrapper[4771]: I1011 10:38:11.643281 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="cluster-policy-controller" Oct 11 10:38:11.643330 master-1 kubenswrapper[4771]: I1011 10:38:11.643296 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerName="kube-controller-manager-cert-syncer" Oct 11 10:38:11.683590 master-1 kubenswrapper[4771]: I1011 10:38:11.683530 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-resource-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0b5de9d609ee1e6c379f71934cb2c3c6\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:11.683708 master-1 kubenswrapper[4771]: I1011 10:38:11.683598 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-cert-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0b5de9d609ee1e6c379f71934cb2c3c6\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:11.784980 master-1 kubenswrapper[4771]: I1011 10:38:11.784527 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-resource-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0b5de9d609ee1e6c379f71934cb2c3c6\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:11.784980 master-1 kubenswrapper[4771]: I1011 10:38:11.784591 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-cert-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0b5de9d609ee1e6c379f71934cb2c3c6\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:11.784980 master-1 kubenswrapper[4771]: I1011 10:38:11.784655 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-resource-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0b5de9d609ee1e6c379f71934cb2c3c6\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:11.784980 master-1 kubenswrapper[4771]: I1011 10:38:11.784713 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-cert-dir\") pod \"kube-controller-manager-master-1\" (UID: \"0b5de9d609ee1e6c379f71934cb2c3c6\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:12.024889 master-1 kubenswrapper[4771]: W1011 10:38:12.024828 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c6c687_55ce_4176_903e_5dadd7371470.slice/crio-295428d0c3674ff9225efcb779e01906679c427acbd4d700762b0d513f5169a8 WatchSource:0}: Error finding container 295428d0c3674ff9225efcb779e01906679c427acbd4d700762b0d513f5169a8: Status 404 returned error can't find the container with id 295428d0c3674ff9225efcb779e01906679c427acbd4d700762b0d513f5169a8 Oct 11 10:38:12.114925 master-1 kubenswrapper[4771]: I1011 10:38:12.114878 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_kube-controller-manager-master-1_0c6dd9eb5bc384e5fbc388e7a2f95c28/kube-controller-manager-cert-syncer/0.log" Oct 11 10:38:12.116592 master-1 kubenswrapper[4771]: I1011 10:38:12.116547 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:12.122927 master-1 kubenswrapper[4771]: I1011 10:38:12.122859 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" oldPodUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" Oct 11 10:38:12.190025 master-1 kubenswrapper[4771]: I1011 10:38:12.188434 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-cert-dir\") pod \"0c6dd9eb5bc384e5fbc388e7a2f95c28\" (UID: \"0c6dd9eb5bc384e5fbc388e7a2f95c28\") " Oct 11 10:38:12.190025 master-1 kubenswrapper[4771]: I1011 10:38:12.188519 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-resource-dir\") pod \"0c6dd9eb5bc384e5fbc388e7a2f95c28\" (UID: \"0c6dd9eb5bc384e5fbc388e7a2f95c28\") " Oct 11 10:38:12.190025 master-1 kubenswrapper[4771]: I1011 10:38:12.188649 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-cert-dir" (OuterVolumeSpecName: "cert-dir") pod "0c6dd9eb5bc384e5fbc388e7a2f95c28" (UID: "0c6dd9eb5bc384e5fbc388e7a2f95c28"). InnerVolumeSpecName "cert-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:38:12.190025 master-1 kubenswrapper[4771]: I1011 10:38:12.188843 4771 reconciler_common.go:293] "Volume detached for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-cert-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.190025 master-1 kubenswrapper[4771]: I1011 10:38:12.188807 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-resource-dir" (OuterVolumeSpecName: "resource-dir") pod "0c6dd9eb5bc384e5fbc388e7a2f95c28" (UID: "0c6dd9eb5bc384e5fbc388e7a2f95c28"). InnerVolumeSpecName "resource-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:38:12.290039 master-1 kubenswrapper[4771]: I1011 10:38:12.289953 4771 reconciler_common.go:293] "Volume detached for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0c6dd9eb5bc384e5fbc388e7a2f95c28-resource-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.294981 master-1 kubenswrapper[4771]: I1011 10:38:12.294936 4771 patch_prober.go:28] interesting pod/kube-controller-manager-guard-master-1 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" start-of-body= Oct 11 10:38:12.295031 master-1 kubenswrapper[4771]: I1011 10:38:12.295004 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podUID="a706deec-9223-4663-9db5-71147d242c34" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" Oct 11 10:38:12.404964 master-1 kubenswrapper[4771]: I1011 10:38:12.404909 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:38:12.445908 master-1 kubenswrapper[4771]: I1011 10:38:12.445816 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" path="/var/lib/kubelet/pods/0c6dd9eb5bc384e5fbc388e7a2f95c28/volumes" Oct 11 10:38:12.459568 master-1 kubenswrapper[4771]: I1011 10:38:12.459513 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-authentication/oauth-openshift-6fccd5ccc-txx8d"] Oct 11 10:38:12.460217 master-1 kubenswrapper[4771]: E1011 10:38:12.460181 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" containerName="oauth-openshift" Oct 11 10:38:12.460217 master-1 kubenswrapper[4771]: I1011 10:38:12.460212 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" containerName="oauth-openshift" Oct 11 10:38:12.460697 master-1 kubenswrapper[4771]: I1011 10:38:12.460663 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" containerName="oauth-openshift" Oct 11 10:38:12.462142 master-1 kubenswrapper[4771]: I1011 10:38:12.462102 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.476703 master-1 kubenswrapper[4771]: I1011 10:38:12.476631 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-authentication/oauth-openshift-6fccd5ccc-txx8d"] Oct 11 10:38:12.520033 master-1 kubenswrapper[4771]: I1011 10:38:12.519981 4771 generic.go:334] "Generic (PLEG): container finished" podID="d61f55f6-6e03-40ca-aa96-cb6ba21c39b4" containerID="5a01f085e3b1c7fb42b9e1bcc547086d47f6a110bdf85f6e451a5f626e8ea9d3" exitCode=0 Oct 11 10:38:12.520213 master-1 kubenswrapper[4771]: I1011 10:38:12.520073 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-5-master-1" event={"ID":"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4","Type":"ContainerDied","Data":"5a01f085e3b1c7fb42b9e1bcc547086d47f6a110bdf85f6e451a5f626e8ea9d3"} Oct 11 10:38:12.521198 master-1 kubenswrapper[4771]: I1011 10:38:12.521154 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" event={"ID":"45c6c687-55ce-4176-903e-5dadd7371470","Type":"ContainerStarted","Data":"295428d0c3674ff9225efcb779e01906679c427acbd4d700762b0d513f5169a8"} Oct 11 10:38:12.522639 master-1 kubenswrapper[4771]: I1011 10:38:12.522584 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" event={"ID":"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8","Type":"ContainerDied","Data":"8893531f4f4ebb704c0ef08b506e4add808eee05f66a87ee0d2eb8eddb5d49b0"} Oct 11 10:38:12.522713 master-1 kubenswrapper[4771]: I1011 10:38:12.522652 4771 scope.go:117] "RemoveContainer" containerID="33d94985a7a913ae973d8eae7753333b859741460e8780a5827eed5110a86a93" Oct 11 10:38:12.522939 master-1 kubenswrapper[4771]: I1011 10:38:12.522926 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-authentication/oauth-openshift-68fb97bcc4-g7k57" Oct 11 10:38:12.524478 master-1 kubenswrapper[4771]: I1011 10:38:12.524454 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-65bb9777fc-66jxg" event={"ID":"6958daf7-e9a7-4151-8e42-851feedec58e","Type":"ContainerStarted","Data":"a368ebd7e5449ec4254f55beacd6ae1c83830fe1507f669e7fc7b33b7cdb82c3"} Oct 11 10:38:12.525031 master-1 kubenswrapper[4771]: I1011 10:38:12.524963 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-console/downloads-65bb9777fc-66jxg" Oct 11 10:38:12.527266 master-1 kubenswrapper[4771]: I1011 10:38:12.527249 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_kube-controller-manager-master-1_0c6dd9eb5bc384e5fbc388e7a2f95c28/kube-controller-manager-cert-syncer/0.log" Oct 11 10:38:12.527343 master-1 kubenswrapper[4771]: I1011 10:38:12.527282 4771 patch_prober.go:28] interesting pod/downloads-65bb9777fc-66jxg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.129.0.66:8080/\": dial tcp 10.129.0.66:8080: connect: connection refused" start-of-body= Oct 11 10:38:12.527419 master-1 kubenswrapper[4771]: I1011 10:38:12.527340 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-65bb9777fc-66jxg" podUID="6958daf7-e9a7-4151-8e42-851feedec58e" containerName="download-server" probeResult="failure" output="Get \"http://10.129.0.66:8080/\": dial tcp 10.129.0.66:8080: connect: connection refused" Oct 11 10:38:12.528045 master-1 kubenswrapper[4771]: I1011 10:38:12.527976 4771 generic.go:334] "Generic (PLEG): container finished" podID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerID="06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1" exitCode=0 Oct 11 10:38:12.528045 master-1 kubenswrapper[4771]: I1011 10:38:12.527998 4771 generic.go:334] "Generic (PLEG): container finished" podID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerID="38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5" exitCode=2 Oct 11 10:38:12.528143 master-1 kubenswrapper[4771]: I1011 10:38:12.528006 4771 generic.go:334] "Generic (PLEG): container finished" podID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerID="c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb" exitCode=0 Oct 11 10:38:12.528214 master-1 kubenswrapper[4771]: I1011 10:38:12.528153 4771 generic.go:334] "Generic (PLEG): container finished" podID="0c6dd9eb5bc384e5fbc388e7a2f95c28" containerID="bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db" exitCode=0 Oct 11 10:38:12.528214 master-1 kubenswrapper[4771]: I1011 10:38:12.528076 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:12.546274 master-1 kubenswrapper[4771]: I1011 10:38:12.546154 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" oldPodUID="0c6dd9eb5bc384e5fbc388e7a2f95c28" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" Oct 11 10:38:12.546961 master-1 kubenswrapper[4771]: I1011 10:38:12.546942 4771 scope.go:117] "RemoveContainer" containerID="06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1" Oct 11 10:38:12.565577 master-1 kubenswrapper[4771]: I1011 10:38:12.565489 4771 scope.go:117] "RemoveContainer" containerID="38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5" Oct 11 10:38:12.573862 master-1 kubenswrapper[4771]: I1011 10:38:12.573788 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-65bb9777fc-66jxg" podStartSLOduration=1.693749492 podStartE2EDuration="29.573762795s" podCreationTimestamp="2025-10-11 10:37:43 +0000 UTC" firstStartedPulling="2025-10-11 10:37:44.384431851 +0000 UTC m=+696.358658292" lastFinishedPulling="2025-10-11 10:38:12.264445154 +0000 UTC m=+724.238671595" observedRunningTime="2025-10-11 10:38:12.570552622 +0000 UTC m=+724.544779073" watchObservedRunningTime="2025-10-11 10:38:12.573762795 +0000 UTC m=+724.547989246" Oct 11 10:38:12.579635 master-1 kubenswrapper[4771]: I1011 10:38:12.579406 4771 scope.go:117] "RemoveContainer" containerID="c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb" Oct 11 10:38:12.592588 master-1 kubenswrapper[4771]: I1011 10:38:12.592557 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-user-template-provider-selection\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-provider-selection\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.592700 master-1 kubenswrapper[4771]: I1011 10:38:12.592613 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-system-router-certs\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-router-certs\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593236 master-1 kubenswrapper[4771]: I1011 10:38:12.593183 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-user-template-login\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-login\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593312 master-1 kubenswrapper[4771]: I1011 10:38:12.593261 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-user-template-error\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-error\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593365 master-1 kubenswrapper[4771]: I1011 10:38:12.593315 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-system-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-service-ca\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593416 master-1 kubenswrapper[4771]: I1011 10:38:12.593347 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-policies\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593416 master-1 kubenswrapper[4771]: I1011 10:38:12.593407 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-system-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-serving-cert\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593489 master-1 kubenswrapper[4771]: I1011 10:38:12.593438 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-system-cliconfig\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-cliconfig\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593489 master-1 kubenswrapper[4771]: I1011 10:38:12.593472 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-dir\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593562 master-1 kubenswrapper[4771]: I1011 10:38:12.593504 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-system-session\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-session\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593595 master-1 kubenswrapper[4771]: I1011 10:38:12.593561 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56hdc\" (UniqueName: \"kubernetes.io/projected/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-kube-api-access-56hdc\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593628 master-1 kubenswrapper[4771]: I1011 10:38:12.593605 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-system-ocp-branding-template\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-ocp-branding-template\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593661 master-1 kubenswrapper[4771]: I1011 10:38:12.593643 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"v4-0-config-system-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-trusted-ca-bundle\") pod \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\" (UID: \"d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8\") " Oct 11 10:38:12.593817 master-1 kubenswrapper[4771]: I1011 10:38:12.593784 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-serving-cert\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.593878 master-1 kubenswrapper[4771]: I1011 10:38:12.593832 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-session\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-session\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.593878 master-1 kubenswrapper[4771]: I1011 10:38:12.593871 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcxh\" (UniqueName: \"kubernetes.io/projected/1ec66eef-540b-4e9a-b63a-02d662224040-kube-api-access-2gcxh\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594024 master-1 kubenswrapper[4771]: I1011 10:38:12.593920 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-audit-policies\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594024 master-1 kubenswrapper[4771]: I1011 10:38:12.593956 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-user-template-login\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-user-template-login\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594105 master-1 kubenswrapper[4771]: I1011 10:38:12.594043 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-user-template-provider-selection\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-user-template-provider-selection\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594105 master-1 kubenswrapper[4771]: I1011 10:38:12.594078 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-service-ca" (OuterVolumeSpecName: "v4-0-config-system-service-ca") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-system-service-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594335 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/1ec66eef-540b-4e9a-b63a-02d662224040-audit-dir\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594381 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-service-ca\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594400 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-router-certs\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-router-certs\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594417 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-trusted-ca-bundle\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594435 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-user-template-error\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-user-template-error\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594451 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-ocp-branding-template\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-ocp-branding-template\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594437 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-cliconfig" (OuterVolumeSpecName: "v4-0-config-system-cliconfig") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-system-cliconfig". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594475 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"v4-0-config-system-cliconfig\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-cliconfig\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594798 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-system-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-service-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.594867 master-1 kubenswrapper[4771]: I1011 10:38:12.594819 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-system-cliconfig\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-cliconfig\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.595629 master-1 kubenswrapper[4771]: I1011 10:38:12.594942 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-policies" (OuterVolumeSpecName: "audit-policies") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "audit-policies". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:38:12.595629 master-1 kubenswrapper[4771]: I1011 10:38:12.594964 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-trusted-ca-bundle" (OuterVolumeSpecName: "v4-0-config-system-trusted-ca-bundle") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-system-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:38:12.595845 master-1 kubenswrapper[4771]: I1011 10:38:12.593798 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:38:12.596396 master-1 kubenswrapper[4771]: I1011 10:38:12.596370 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-router-certs" (OuterVolumeSpecName: "v4-0-config-system-router-certs") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-system-router-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:38:12.596544 master-1 kubenswrapper[4771]: I1011 10:38:12.596522 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-provider-selection" (OuterVolumeSpecName: "v4-0-config-user-template-provider-selection") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-user-template-provider-selection". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:38:12.596817 master-1 kubenswrapper[4771]: I1011 10:38:12.596711 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-login" (OuterVolumeSpecName: "v4-0-config-user-template-login") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-user-template-login". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:38:12.597720 master-1 kubenswrapper[4771]: I1011 10:38:12.597633 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-session" (OuterVolumeSpecName: "v4-0-config-system-session") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-system-session". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:38:12.598060 master-1 kubenswrapper[4771]: I1011 10:38:12.598031 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-serving-cert" (OuterVolumeSpecName: "v4-0-config-system-serving-cert") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-system-serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:38:12.599510 master-1 kubenswrapper[4771]: I1011 10:38:12.599475 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-ocp-branding-template" (OuterVolumeSpecName: "v4-0-config-system-ocp-branding-template") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-system-ocp-branding-template". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:38:12.601434 master-1 kubenswrapper[4771]: I1011 10:38:12.601391 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-error" (OuterVolumeSpecName: "v4-0-config-user-template-error") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "v4-0-config-user-template-error". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:38:12.603413 master-1 kubenswrapper[4771]: I1011 10:38:12.603385 4771 scope.go:117] "RemoveContainer" containerID="bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db" Oct 11 10:38:12.605169 master-1 kubenswrapper[4771]: I1011 10:38:12.605130 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-kube-api-access-56hdc" (OuterVolumeSpecName: "kube-api-access-56hdc") pod "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" (UID: "d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8"). InnerVolumeSpecName "kube-api-access-56hdc". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:38:12.619444 master-1 kubenswrapper[4771]: I1011 10:38:12.619326 4771 scope.go:117] "RemoveContainer" containerID="06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1" Oct 11 10:38:12.619944 master-1 kubenswrapper[4771]: E1011 10:38:12.619841 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1\": container with ID starting with 06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1 not found: ID does not exist" containerID="06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1" Oct 11 10:38:12.620010 master-1 kubenswrapper[4771]: I1011 10:38:12.619951 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1"} err="failed to get container status \"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1\": rpc error: code = NotFound desc = could not find container \"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1\": container with ID starting with 06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1 not found: ID does not exist" Oct 11 10:38:12.620010 master-1 kubenswrapper[4771]: I1011 10:38:12.620002 4771 scope.go:117] "RemoveContainer" containerID="38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5" Oct 11 10:38:12.620584 master-1 kubenswrapper[4771]: E1011 10:38:12.620464 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5\": container with ID starting with 38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5 not found: ID does not exist" containerID="38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5" Oct 11 10:38:12.620584 master-1 kubenswrapper[4771]: I1011 10:38:12.620506 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5"} err="failed to get container status \"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5\": rpc error: code = NotFound desc = could not find container \"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5\": container with ID starting with 38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5 not found: ID does not exist" Oct 11 10:38:12.620584 master-1 kubenswrapper[4771]: I1011 10:38:12.620534 4771 scope.go:117] "RemoveContainer" containerID="c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb" Oct 11 10:38:12.620828 master-1 kubenswrapper[4771]: E1011 10:38:12.620798 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb\": container with ID starting with c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb not found: ID does not exist" containerID="c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb" Oct 11 10:38:12.620913 master-1 kubenswrapper[4771]: I1011 10:38:12.620825 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb"} err="failed to get container status \"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb\": rpc error: code = NotFound desc = could not find container \"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb\": container with ID starting with c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb not found: ID does not exist" Oct 11 10:38:12.620913 master-1 kubenswrapper[4771]: I1011 10:38:12.620841 4771 scope.go:117] "RemoveContainer" containerID="bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db" Oct 11 10:38:12.621131 master-1 kubenswrapper[4771]: E1011 10:38:12.621101 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db\": container with ID starting with bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db not found: ID does not exist" containerID="bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db" Oct 11 10:38:12.621184 master-1 kubenswrapper[4771]: I1011 10:38:12.621127 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db"} err="failed to get container status \"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db\": rpc error: code = NotFound desc = could not find container \"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db\": container with ID starting with bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db not found: ID does not exist" Oct 11 10:38:12.621184 master-1 kubenswrapper[4771]: I1011 10:38:12.621145 4771 scope.go:117] "RemoveContainer" containerID="06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1" Oct 11 10:38:12.621429 master-1 kubenswrapper[4771]: I1011 10:38:12.621406 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1"} err="failed to get container status \"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1\": rpc error: code = NotFound desc = could not find container \"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1\": container with ID starting with 06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1 not found: ID does not exist" Oct 11 10:38:12.621495 master-1 kubenswrapper[4771]: I1011 10:38:12.621430 4771 scope.go:117] "RemoveContainer" containerID="38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5" Oct 11 10:38:12.621947 master-1 kubenswrapper[4771]: I1011 10:38:12.621913 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5"} err="failed to get container status \"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5\": rpc error: code = NotFound desc = could not find container \"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5\": container with ID starting with 38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5 not found: ID does not exist" Oct 11 10:38:12.621947 master-1 kubenswrapper[4771]: I1011 10:38:12.621937 4771 scope.go:117] "RemoveContainer" containerID="c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb" Oct 11 10:38:12.622267 master-1 kubenswrapper[4771]: I1011 10:38:12.622243 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb"} err="failed to get container status \"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb\": rpc error: code = NotFound desc = could not find container \"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb\": container with ID starting with c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb not found: ID does not exist" Oct 11 10:38:12.622267 master-1 kubenswrapper[4771]: I1011 10:38:12.622261 4771 scope.go:117] "RemoveContainer" containerID="bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db" Oct 11 10:38:12.622579 master-1 kubenswrapper[4771]: I1011 10:38:12.622560 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db"} err="failed to get container status \"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db\": rpc error: code = NotFound desc = could not find container \"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db\": container with ID starting with bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db not found: ID does not exist" Oct 11 10:38:12.622579 master-1 kubenswrapper[4771]: I1011 10:38:12.622578 4771 scope.go:117] "RemoveContainer" containerID="06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1" Oct 11 10:38:12.627053 master-1 kubenswrapper[4771]: I1011 10:38:12.627011 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1"} err="failed to get container status \"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1\": rpc error: code = NotFound desc = could not find container \"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1\": container with ID starting with 06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1 not found: ID does not exist" Oct 11 10:38:12.627053 master-1 kubenswrapper[4771]: I1011 10:38:12.627042 4771 scope.go:117] "RemoveContainer" containerID="38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5" Oct 11 10:38:12.627340 master-1 kubenswrapper[4771]: I1011 10:38:12.627310 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5"} err="failed to get container status \"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5\": rpc error: code = NotFound desc = could not find container \"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5\": container with ID starting with 38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5 not found: ID does not exist" Oct 11 10:38:12.627412 master-1 kubenswrapper[4771]: I1011 10:38:12.627340 4771 scope.go:117] "RemoveContainer" containerID="c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb" Oct 11 10:38:12.627754 master-1 kubenswrapper[4771]: I1011 10:38:12.627682 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb"} err="failed to get container status \"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb\": rpc error: code = NotFound desc = could not find container \"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb\": container with ID starting with c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb not found: ID does not exist" Oct 11 10:38:12.627754 master-1 kubenswrapper[4771]: I1011 10:38:12.627734 4771 scope.go:117] "RemoveContainer" containerID="bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db" Oct 11 10:38:12.628268 master-1 kubenswrapper[4771]: I1011 10:38:12.628210 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db"} err="failed to get container status \"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db\": rpc error: code = NotFound desc = could not find container \"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db\": container with ID starting with bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db not found: ID does not exist" Oct 11 10:38:12.628349 master-1 kubenswrapper[4771]: I1011 10:38:12.628263 4771 scope.go:117] "RemoveContainer" containerID="06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1" Oct 11 10:38:12.628674 master-1 kubenswrapper[4771]: I1011 10:38:12.628646 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1"} err="failed to get container status \"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1\": rpc error: code = NotFound desc = could not find container \"06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1\": container with ID starting with 06f70e119ed613e4288345adac3908921043a544ab2e25638aba6006871402c1 not found: ID does not exist" Oct 11 10:38:12.628764 master-1 kubenswrapper[4771]: I1011 10:38:12.628674 4771 scope.go:117] "RemoveContainer" containerID="38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5" Oct 11 10:38:12.629151 master-1 kubenswrapper[4771]: I1011 10:38:12.629056 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5"} err="failed to get container status \"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5\": rpc error: code = NotFound desc = could not find container \"38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5\": container with ID starting with 38defd4315f781404a7b26e3ce886c78170b46fa258e5d8165a7bbf19f839fb5 not found: ID does not exist" Oct 11 10:38:12.629151 master-1 kubenswrapper[4771]: I1011 10:38:12.629082 4771 scope.go:117] "RemoveContainer" containerID="c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb" Oct 11 10:38:12.629404 master-1 kubenswrapper[4771]: I1011 10:38:12.629379 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb"} err="failed to get container status \"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb\": rpc error: code = NotFound desc = could not find container \"c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb\": container with ID starting with c70e6cc8f9016d52eff2617265ed98972e2a524994971b81a6c9f6c45650cecb not found: ID does not exist" Oct 11 10:38:12.629475 master-1 kubenswrapper[4771]: I1011 10:38:12.629406 4771 scope.go:117] "RemoveContainer" containerID="bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db" Oct 11 10:38:12.629721 master-1 kubenswrapper[4771]: I1011 10:38:12.629696 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db"} err="failed to get container status \"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db\": rpc error: code = NotFound desc = could not find container \"bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db\": container with ID starting with bc03748902a6d44f293d452c1632075da1fc534b6acb87323d789e53beede2db not found: ID does not exist" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695395 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-ocp-branding-template\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-ocp-branding-template\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695457 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-cliconfig\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-cliconfig\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695495 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-serving-cert\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695520 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-session\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-session\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695544 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcxh\" (UniqueName: \"kubernetes.io/projected/1ec66eef-540b-4e9a-b63a-02d662224040-kube-api-access-2gcxh\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695567 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-audit-policies\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695590 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-user-template-login\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-user-template-login\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695628 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-user-template-provider-selection\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-user-template-provider-selection\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695682 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/1ec66eef-540b-4e9a-b63a-02d662224040-audit-dir\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695785 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-service-ca\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695813 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-router-certs\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-router-certs\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695890 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-system-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-trusted-ca-bundle\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.695960 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"v4-0-config-user-template-error\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-user-template-error\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.696014 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-system-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.696031 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-user-template-provider-selection\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-provider-selection\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.696761 master-1 kubenswrapper[4771]: I1011 10:38:12.696031 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/1ec66eef-540b-4e9a-b63a-02d662224040-audit-dir\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696047 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-system-router-certs\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-router-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696149 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-user-template-login\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-login\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696166 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-user-template-error\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-user-template-error\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696180 4771 reconciler_common.go:293] "Volume detached for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-policies\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696194 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-system-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696206 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696217 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-system-session\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-session\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696229 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-56hdc\" (UniqueName: \"kubernetes.io/projected/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-kube-api-access-56hdc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696281 4771 reconciler_common.go:293] "Volume detached for volume \"v4-0-config-system-ocp-branding-template\" (UniqueName: \"kubernetes.io/secret/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8-v4-0-config-system-ocp-branding-template\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696416 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-cliconfig\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-cliconfig\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.696551 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-audit-policies\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.697402 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-service-ca\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.697982 master-1 kubenswrapper[4771]: I1011 10:38:12.697430 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-trusted-ca-bundle\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.700529 master-1 kubenswrapper[4771]: I1011 10:38:12.700458 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-serving-cert\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.700701 master-1 kubenswrapper[4771]: I1011 10:38:12.700676 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-router-certs\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-router-certs\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.701024 master-1 kubenswrapper[4771]: I1011 10:38:12.700924 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-ocp-branding-template\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-ocp-branding-template\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.701024 master-1 kubenswrapper[4771]: I1011 10:38:12.700962 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-user-template-error\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-user-template-error\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.701186 master-1 kubenswrapper[4771]: I1011 10:38:12.701042 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-system-session\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-system-session\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.701287 master-1 kubenswrapper[4771]: I1011 10:38:12.701267 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-user-template-login\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-user-template-login\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.702062 master-1 kubenswrapper[4771]: I1011 10:38:12.702007 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"v4-0-config-user-template-provider-selection\" (UniqueName: \"kubernetes.io/secret/1ec66eef-540b-4e9a-b63a-02d662224040-v4-0-config-user-template-provider-selection\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.714173 master-1 kubenswrapper[4771]: I1011 10:38:12.714139 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcxh\" (UniqueName: \"kubernetes.io/projected/1ec66eef-540b-4e9a-b63a-02d662224040-kube-api-access-2gcxh\") pod \"oauth-openshift-6fccd5ccc-txx8d\" (UID: \"1ec66eef-540b-4e9a-b63a-02d662224040\") " pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.718389 master-1 kubenswrapper[4771]: I1011 10:38:12.718250 4771 scope.go:117] "RemoveContainer" containerID="ed4ea2c827d3365e80a136d8fc9c70fdea44747628fc9e1b440208d196a14d73" Oct 11 10:38:12.742518 master-1 kubenswrapper[4771]: I1011 10:38:12.742422 4771 scope.go:117] "RemoveContainer" containerID="5314d6ef2281ac080baefb268e1b24e3959c52d75eecf8bba9e60d0238801c00" Oct 11 10:38:12.761538 master-1 kubenswrapper[4771]: I1011 10:38:12.761471 4771 scope.go:117] "RemoveContainer" containerID="9b7973318d321c4747b9166204be01b90470f6b7ff6c1031063eb5d24ec05b0e" Oct 11 10:38:12.796720 master-1 kubenswrapper[4771]: I1011 10:38:12.796276 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:12.856263 master-1 kubenswrapper[4771]: I1011 10:38:12.856205 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-authentication/oauth-openshift-68fb97bcc4-g7k57"] Oct 11 10:38:12.867854 master-1 kubenswrapper[4771]: I1011 10:38:12.867809 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-authentication/oauth-openshift-68fb97bcc4-g7k57"] Oct 11 10:38:13.537201 master-1 kubenswrapper[4771]: I1011 10:38:13.537087 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" event={"ID":"45c6c687-55ce-4176-903e-5dadd7371470","Type":"ContainerStarted","Data":"0ea4367e2967b4a12600d203935e4ed28870df606f1322b3a2afbe5f5edbccd3"} Oct 11 10:38:13.540901 master-1 kubenswrapper[4771]: I1011 10:38:13.540846 4771 patch_prober.go:28] interesting pod/downloads-65bb9777fc-66jxg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.129.0.66:8080/\": dial tcp 10.129.0.66:8080: connect: connection refused" start-of-body= Oct 11 10:38:13.541000 master-1 kubenswrapper[4771]: I1011 10:38:13.540929 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-65bb9777fc-66jxg" podUID="6958daf7-e9a7-4151-8e42-851feedec58e" containerName="download-server" probeResult="failure" output="Get \"http://10.129.0.66:8080/\": dial tcp 10.129.0.66:8080: connect: connection refused" Oct 11 10:38:13.564616 master-1 kubenswrapper[4771]: I1011 10:38:13.561577 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-85df6bdd68-48crk" podStartSLOduration=12.381453325 podStartE2EDuration="13.561555717s" podCreationTimestamp="2025-10-11 10:38:00 +0000 UTC" firstStartedPulling="2025-10-11 10:38:12.027620572 +0000 UTC m=+724.001847023" lastFinishedPulling="2025-10-11 10:38:13.207722954 +0000 UTC m=+725.181949415" observedRunningTime="2025-10-11 10:38:13.559694743 +0000 UTC m=+725.533921224" watchObservedRunningTime="2025-10-11 10:38:13.561555717 +0000 UTC m=+725.535782188" Oct 11 10:38:13.638633 master-1 kubenswrapper[4771]: I1011 10:38:13.638503 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-authentication/oauth-openshift-6fccd5ccc-txx8d"] Oct 11 10:38:13.872776 master-1 kubenswrapper[4771]: I1011 10:38:13.872371 4771 patch_prober.go:28] interesting pod/downloads-65bb9777fc-66jxg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.129.0.66:8080/\": dial tcp 10.129.0.66:8080: connect: connection refused" start-of-body= Oct 11 10:38:13.872776 master-1 kubenswrapper[4771]: I1011 10:38:13.872430 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-65bb9777fc-66jxg" podUID="6958daf7-e9a7-4151-8e42-851feedec58e" containerName="download-server" probeResult="failure" output="Get \"http://10.129.0.66:8080/\": dial tcp 10.129.0.66:8080: connect: connection refused" Oct 11 10:38:13.872776 master-1 kubenswrapper[4771]: I1011 10:38:13.872557 4771 patch_prober.go:28] interesting pod/downloads-65bb9777fc-66jxg container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.129.0.66:8080/\": dial tcp 10.129.0.66:8080: connect: connection refused" start-of-body= Oct 11 10:38:13.872776 master-1 kubenswrapper[4771]: I1011 10:38:13.872655 4771 prober.go:107] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-65bb9777fc-66jxg" podUID="6958daf7-e9a7-4151-8e42-851feedec58e" containerName="download-server" probeResult="failure" output="Get \"http://10.129.0.66:8080/\": dial tcp 10.129.0.66:8080: connect: connection refused" Oct 11 10:38:13.913226 master-1 kubenswrapper[4771]: I1011 10:38:13.913179 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:38:14.024814 master-1 kubenswrapper[4771]: I1011 10:38:14.024640 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kubelet-dir\") pod \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " Oct 11 10:38:14.025159 master-1 kubenswrapper[4771]: I1011 10:38:14.024756 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "d61f55f6-6e03-40ca-aa96-cb6ba21c39b4" (UID: "d61f55f6-6e03-40ca-aa96-cb6ba21c39b4"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:38:14.025986 master-1 kubenswrapper[4771]: I1011 10:38:14.025348 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kube-api-access\") pod \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " Oct 11 10:38:14.026298 master-1 kubenswrapper[4771]: I1011 10:38:14.026272 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-var-lock\") pod \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\" (UID: \"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4\") " Oct 11 10:38:14.026611 master-1 kubenswrapper[4771]: I1011 10:38:14.026434 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-var-lock" (OuterVolumeSpecName: "var-lock") pod "d61f55f6-6e03-40ca-aa96-cb6ba21c39b4" (UID: "d61f55f6-6e03-40ca-aa96-cb6ba21c39b4"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:38:14.027317 master-1 kubenswrapper[4771]: I1011 10:38:14.027278 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:14.027672 master-1 kubenswrapper[4771]: I1011 10:38:14.027639 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:14.029145 master-1 kubenswrapper[4771]: I1011 10:38:14.029070 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "d61f55f6-6e03-40ca-aa96-cb6ba21c39b4" (UID: "d61f55f6-6e03-40ca-aa96-cb6ba21c39b4"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:38:14.129505 master-1 kubenswrapper[4771]: I1011 10:38:14.129309 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/d61f55f6-6e03-40ca-aa96-cb6ba21c39b4-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:14.447683 master-1 kubenswrapper[4771]: I1011 10:38:14.447516 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8" path="/var/lib/kubelet/pods/d2a2fcbb-6cfa-4d22-ba29-7f08edccdee8/volumes" Oct 11 10:38:14.549126 master-1 kubenswrapper[4771]: I1011 10:38:14.548609 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" event={"ID":"1ec66eef-540b-4e9a-b63a-02d662224040","Type":"ContainerStarted","Data":"fcd7151178a5f5be90381dc50d2b0387c668e8ffa4d74ae87b04086b2bf41165"} Oct 11 10:38:14.549126 master-1 kubenswrapper[4771]: I1011 10:38:14.548686 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" event={"ID":"1ec66eef-540b-4e9a-b63a-02d662224040","Type":"ContainerStarted","Data":"9444a4249ba1568aacf3e1abe26947d48b9cfe0ff588bf5939efb08250e848de"} Oct 11 10:38:14.549126 master-1 kubenswrapper[4771]: I1011 10:38:14.548877 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:14.552489 master-1 kubenswrapper[4771]: I1011 10:38:14.552420 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-5-master-1" event={"ID":"d61f55f6-6e03-40ca-aa96-cb6ba21c39b4","Type":"ContainerDied","Data":"d51941eb09c3e4a595d1102ccd7dab8966ed8157ff482e09bff14b3b01ba141c"} Oct 11 10:38:14.552623 master-1 kubenswrapper[4771]: I1011 10:38:14.552499 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d51941eb09c3e4a595d1102ccd7dab8966ed8157ff482e09bff14b3b01ba141c" Oct 11 10:38:14.552891 master-1 kubenswrapper[4771]: I1011 10:38:14.552850 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-5-master-1" Oct 11 10:38:14.558018 master-1 kubenswrapper[4771]: I1011 10:38:14.557950 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" Oct 11 10:38:14.614730 master-1 kubenswrapper[4771]: I1011 10:38:14.614600 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-authentication/oauth-openshift-6fccd5ccc-txx8d" podStartSLOduration=14.614571878 podStartE2EDuration="14.614571878s" podCreationTimestamp="2025-10-11 10:38:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:38:14.584668881 +0000 UTC m=+726.558895362" watchObservedRunningTime="2025-10-11 10:38:14.614571878 +0000 UTC m=+726.588798349" Oct 11 10:38:17.267538 master-1 kubenswrapper[4771]: I1011 10:38:17.267436 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d46fcc5c6-bhfmd" Oct 11 10:38:17.293976 master-1 kubenswrapper[4771]: I1011 10:38:17.293878 4771 patch_prober.go:28] interesting pod/kube-controller-manager-guard-master-1 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" start-of-body= Oct 11 10:38:17.293976 master-1 kubenswrapper[4771]: I1011 10:38:17.293959 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podUID="a706deec-9223-4663-9db5-71147d242c34" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" Oct 11 10:38:18.406717 master-1 kubenswrapper[4771]: I1011 10:38:18.406539 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-console/console-57bccbfdf6-l962w" podUID="a7c05954-4353-4cd1-9130-7fcb832a0493" containerName="console" containerID="cri-o://ac4cbca778491aecc2f71b3dc29feeec7e7dede29f2bc161a327a52e374f391b" gracePeriod=15 Oct 11 10:38:18.584825 master-1 kubenswrapper[4771]: I1011 10:38:18.584736 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57bccbfdf6-l962w_a7c05954-4353-4cd1-9130-7fcb832a0493/console/0.log" Oct 11 10:38:18.585093 master-1 kubenswrapper[4771]: I1011 10:38:18.584852 4771 generic.go:334] "Generic (PLEG): container finished" podID="a7c05954-4353-4cd1-9130-7fcb832a0493" containerID="ac4cbca778491aecc2f71b3dc29feeec7e7dede29f2bc161a327a52e374f391b" exitCode=2 Oct 11 10:38:18.585093 master-1 kubenswrapper[4771]: I1011 10:38:18.584913 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bccbfdf6-l962w" event={"ID":"a7c05954-4353-4cd1-9130-7fcb832a0493","Type":"ContainerDied","Data":"ac4cbca778491aecc2f71b3dc29feeec7e7dede29f2bc161a327a52e374f391b"} Oct 11 10:38:18.640905 master-1 kubenswrapper[4771]: I1011 10:38:18.640820 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/installer-4-master-1"] Oct 11 10:38:18.641196 master-1 kubenswrapper[4771]: E1011 10:38:18.641147 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d61f55f6-6e03-40ca-aa96-cb6ba21c39b4" containerName="installer" Oct 11 10:38:18.641196 master-1 kubenswrapper[4771]: I1011 10:38:18.641179 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d61f55f6-6e03-40ca-aa96-cb6ba21c39b4" containerName="installer" Oct 11 10:38:18.641452 master-1 kubenswrapper[4771]: I1011 10:38:18.641405 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d61f55f6-6e03-40ca-aa96-cb6ba21c39b4" containerName="installer" Oct 11 10:38:18.642407 master-1 kubenswrapper[4771]: I1011 10:38:18.642313 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:18.646436 master-1 kubenswrapper[4771]: I1011 10:38:18.646342 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-kube-apiserver"/"installer-sa-dockercfg-t28rg" Oct 11 10:38:18.709780 master-1 kubenswrapper[4771]: I1011 10:38:18.709602 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-var-lock\") pod \"installer-4-master-1\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:18.709780 master-1 kubenswrapper[4771]: I1011 10:38:18.709674 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-kubelet-dir\") pod \"installer-4-master-1\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:18.709780 master-1 kubenswrapper[4771]: I1011 10:38:18.709708 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/66232733-dcfb-4320-a372-ce05d7d777d9-kube-api-access\") pod \"installer-4-master-1\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:18.811507 master-1 kubenswrapper[4771]: I1011 10:38:18.811348 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-var-lock\") pod \"installer-4-master-1\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:18.811507 master-1 kubenswrapper[4771]: I1011 10:38:18.811462 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-kubelet-dir\") pod \"installer-4-master-1\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:18.811507 master-1 kubenswrapper[4771]: I1011 10:38:18.811519 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/66232733-dcfb-4320-a372-ce05d7d777d9-kube-api-access\") pod \"installer-4-master-1\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:18.811969 master-1 kubenswrapper[4771]: I1011 10:38:18.811524 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-var-lock\") pod \"installer-4-master-1\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:18.811969 master-1 kubenswrapper[4771]: I1011 10:38:18.811603 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-kubelet-dir\") pod \"installer-4-master-1\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:18.962163 master-1 kubenswrapper[4771]: I1011 10:38:18.962033 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57bccbfdf6-l962w_a7c05954-4353-4cd1-9130-7fcb832a0493/console/0.log" Oct 11 10:38:18.962163 master-1 kubenswrapper[4771]: I1011 10:38:18.962141 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:38:18.975892 master-1 kubenswrapper[4771]: I1011 10:38:18.975830 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-4-master-1"] Oct 11 10:38:19.014385 master-1 kubenswrapper[4771]: I1011 10:38:19.014277 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-oauth-serving-cert\") pod \"a7c05954-4353-4cd1-9130-7fcb832a0493\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " Oct 11 10:38:19.014385 master-1 kubenswrapper[4771]: I1011 10:38:19.014348 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-console-config\") pod \"a7c05954-4353-4cd1-9130-7fcb832a0493\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " Oct 11 10:38:19.014610 master-1 kubenswrapper[4771]: I1011 10:38:19.014438 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngk7c\" (UniqueName: \"kubernetes.io/projected/a7c05954-4353-4cd1-9130-7fcb832a0493-kube-api-access-ngk7c\") pod \"a7c05954-4353-4cd1-9130-7fcb832a0493\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " Oct 11 10:38:19.014610 master-1 kubenswrapper[4771]: I1011 10:38:19.014496 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-serving-cert\") pod \"a7c05954-4353-4cd1-9130-7fcb832a0493\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " Oct 11 10:38:19.014610 master-1 kubenswrapper[4771]: I1011 10:38:19.014579 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-oauth-config\") pod \"a7c05954-4353-4cd1-9130-7fcb832a0493\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " Oct 11 10:38:19.015969 master-1 kubenswrapper[4771]: I1011 10:38:19.014669 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-service-ca\") pod \"a7c05954-4353-4cd1-9130-7fcb832a0493\" (UID: \"a7c05954-4353-4cd1-9130-7fcb832a0493\") " Oct 11 10:38:19.015969 master-1 kubenswrapper[4771]: I1011 10:38:19.014761 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7c05954-4353-4cd1-9130-7fcb832a0493" (UID: "a7c05954-4353-4cd1-9130-7fcb832a0493"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:38:19.015969 master-1 kubenswrapper[4771]: I1011 10:38:19.015072 4771 reconciler_common.go:293] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-oauth-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:19.015969 master-1 kubenswrapper[4771]: I1011 10:38:19.015545 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-console-config" (OuterVolumeSpecName: "console-config") pod "a7c05954-4353-4cd1-9130-7fcb832a0493" (UID: "a7c05954-4353-4cd1-9130-7fcb832a0493"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:38:19.015969 master-1 kubenswrapper[4771]: I1011 10:38:19.015643 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7c05954-4353-4cd1-9130-7fcb832a0493" (UID: "a7c05954-4353-4cd1-9130-7fcb832a0493"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:38:19.018735 master-1 kubenswrapper[4771]: I1011 10:38:19.018659 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7c05954-4353-4cd1-9130-7fcb832a0493" (UID: "a7c05954-4353-4cd1-9130-7fcb832a0493"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:38:19.018984 master-1 kubenswrapper[4771]: I1011 10:38:19.018927 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c05954-4353-4cd1-9130-7fcb832a0493-kube-api-access-ngk7c" (OuterVolumeSpecName: "kube-api-access-ngk7c") pod "a7c05954-4353-4cd1-9130-7fcb832a0493" (UID: "a7c05954-4353-4cd1-9130-7fcb832a0493"). InnerVolumeSpecName "kube-api-access-ngk7c". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:38:19.019499 master-1 kubenswrapper[4771]: I1011 10:38:19.019438 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7c05954-4353-4cd1-9130-7fcb832a0493" (UID: "a7c05954-4353-4cd1-9130-7fcb832a0493"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:38:19.116994 master-1 kubenswrapper[4771]: I1011 10:38:19.116891 4771 reconciler_common.go:293] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-service-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:19.116994 master-1 kubenswrapper[4771]: I1011 10:38:19.116952 4771 reconciler_common.go:293] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7c05954-4353-4cd1-9130-7fcb832a0493-console-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:19.116994 master-1 kubenswrapper[4771]: I1011 10:38:19.116975 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-ngk7c\" (UniqueName: \"kubernetes.io/projected/a7c05954-4353-4cd1-9130-7fcb832a0493-kube-api-access-ngk7c\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:19.116994 master-1 kubenswrapper[4771]: I1011 10:38:19.116997 4771 reconciler_common.go:293] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:19.117465 master-1 kubenswrapper[4771]: I1011 10:38:19.117018 4771 reconciler_common.go:293] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7c05954-4353-4cd1-9130-7fcb832a0493-console-oauth-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:19.593441 master-1 kubenswrapper[4771]: I1011 10:38:19.593317 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57bccbfdf6-l962w_a7c05954-4353-4cd1-9130-7fcb832a0493/console/0.log" Oct 11 10:38:19.594530 master-1 kubenswrapper[4771]: I1011 10:38:19.593462 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bccbfdf6-l962w" event={"ID":"a7c05954-4353-4cd1-9130-7fcb832a0493","Type":"ContainerDied","Data":"8b80ae3136f1889aef88e388582b0fe2c7b18eb0654ca449f848a798f30b4031"} Oct 11 10:38:19.594530 master-1 kubenswrapper[4771]: I1011 10:38:19.593513 4771 scope.go:117] "RemoveContainer" containerID="ac4cbca778491aecc2f71b3dc29feeec7e7dede29f2bc161a327a52e374f391b" Oct 11 10:38:19.594530 master-1 kubenswrapper[4771]: I1011 10:38:19.593572 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bccbfdf6-l962w" Oct 11 10:38:20.033886 master-1 kubenswrapper[4771]: I1011 10:38:20.033652 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/66232733-dcfb-4320-a372-ce05d7d777d9-kube-api-access\") pod \"installer-4-master-1\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:20.162179 master-1 kubenswrapper[4771]: I1011 10:38:20.162067 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:20.571518 master-1 kubenswrapper[4771]: I1011 10:38:20.571391 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57bccbfdf6-l962w"] Oct 11 10:38:20.751126 master-1 kubenswrapper[4771]: I1011 10:38:20.750932 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-console/console-775ff6c4fc-csp4z"] Oct 11 10:38:20.752228 master-1 kubenswrapper[4771]: E1011 10:38:20.752196 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a7c05954-4353-4cd1-9130-7fcb832a0493" containerName="console" Oct 11 10:38:20.752390 master-1 kubenswrapper[4771]: I1011 10:38:20.752337 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c05954-4353-4cd1-9130-7fcb832a0493" containerName="console" Oct 11 10:38:20.752656 master-1 kubenswrapper[4771]: I1011 10:38:20.752632 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a7c05954-4353-4cd1-9130-7fcb832a0493" containerName="console" Oct 11 10:38:20.753537 master-1 kubenswrapper[4771]: I1011 10:38:20.753504 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.757192 master-1 kubenswrapper[4771]: I1011 10:38:20.757113 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"service-ca" Oct 11 10:38:20.762014 master-1 kubenswrapper[4771]: I1011 10:38:20.761968 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"console-dockercfg-8wmjp" Oct 11 10:38:20.762157 master-1 kubenswrapper[4771]: I1011 10:38:20.762028 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"console-config" Oct 11 10:38:20.762578 master-1 kubenswrapper[4771]: I1011 10:38:20.762536 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"console-oauth-config" Oct 11 10:38:20.762697 master-1 kubenswrapper[4771]: I1011 10:38:20.762649 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"oauth-serving-cert" Oct 11 10:38:20.763155 master-1 kubenswrapper[4771]: I1011 10:38:20.763106 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"console-serving-cert" Oct 11 10:38:20.769207 master-1 kubenswrapper[4771]: I1011 10:38:20.769169 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"trusted-ca-bundle" Oct 11 10:38:20.849245 master-1 kubenswrapper[4771]: I1011 10:38:20.849151 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-service-ca\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.849574 master-1 kubenswrapper[4771]: I1011 10:38:20.849254 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-oauth-serving-cert\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.849574 master-1 kubenswrapper[4771]: I1011 10:38:20.849300 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-serving-cert\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.849574 master-1 kubenswrapper[4771]: I1011 10:38:20.849339 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmzlm\" (UniqueName: \"kubernetes.io/projected/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-kube-api-access-mmzlm\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.849809 master-1 kubenswrapper[4771]: I1011 10:38:20.849546 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-oauth-config\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.849809 master-1 kubenswrapper[4771]: I1011 10:38:20.849628 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-trusted-ca-bundle\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.850028 master-1 kubenswrapper[4771]: I1011 10:38:20.849976 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-config\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.925573 master-1 kubenswrapper[4771]: I1011 10:38:20.925444 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-775ff6c4fc-csp4z"] Oct 11 10:38:20.926931 master-1 kubenswrapper[4771]: I1011 10:38:20.926870 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57bccbfdf6-l962w"] Oct 11 10:38:20.954019 master-1 kubenswrapper[4771]: I1011 10:38:20.953914 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-mmzlm\" (UniqueName: \"kubernetes.io/projected/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-kube-api-access-mmzlm\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.954019 master-1 kubenswrapper[4771]: I1011 10:38:20.954026 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-oauth-config\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.954610 master-1 kubenswrapper[4771]: I1011 10:38:20.954106 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-trusted-ca-bundle\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.954610 master-1 kubenswrapper[4771]: I1011 10:38:20.954258 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-config\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.954610 master-1 kubenswrapper[4771]: I1011 10:38:20.954296 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-service-ca\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.954610 master-1 kubenswrapper[4771]: I1011 10:38:20.954337 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-oauth-serving-cert\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.955957 master-1 kubenswrapper[4771]: I1011 10:38:20.955892 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-oauth-serving-cert\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.956108 master-1 kubenswrapper[4771]: I1011 10:38:20.955962 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-service-ca\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.956108 master-1 kubenswrapper[4771]: I1011 10:38:20.956089 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-trusted-ca-bundle\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.956296 master-1 kubenswrapper[4771]: I1011 10:38:20.956203 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-serving-cert\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.957725 master-1 kubenswrapper[4771]: I1011 10:38:20.957689 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-config\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.962438 master-1 kubenswrapper[4771]: I1011 10:38:20.962030 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-serving-cert\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:20.965055 master-1 kubenswrapper[4771]: I1011 10:38:20.964997 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-oauth-config\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:21.302943 master-1 kubenswrapper[4771]: I1011 10:38:21.302704 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-4-master-1"] Oct 11 10:38:21.309000 master-1 kubenswrapper[4771]: W1011 10:38:21.308910 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod66232733_dcfb_4320_a372_ce05d7d777d9.slice/crio-210c1e83cdd57074a16669f3f9ab89020bba5fd50626a163674b365ff40935aa WatchSource:0}: Error finding container 210c1e83cdd57074a16669f3f9ab89020bba5fd50626a163674b365ff40935aa: Status 404 returned error can't find the container with id 210c1e83cdd57074a16669f3f9ab89020bba5fd50626a163674b365ff40935aa Oct 11 10:38:21.613994 master-1 kubenswrapper[4771]: I1011 10:38:21.613877 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-4-master-1" event={"ID":"66232733-dcfb-4320-a372-ce05d7d777d9","Type":"ContainerStarted","Data":"210c1e83cdd57074a16669f3f9ab89020bba5fd50626a163674b365ff40935aa"} Oct 11 10:38:22.293665 master-1 kubenswrapper[4771]: I1011 10:38:22.293593 4771 patch_prober.go:28] interesting pod/kube-controller-manager-guard-master-1 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" start-of-body= Oct 11 10:38:22.293665 master-1 kubenswrapper[4771]: I1011 10:38:22.293660 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podUID="a706deec-9223-4663-9db5-71147d242c34" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" Oct 11 10:38:22.294406 master-1 kubenswrapper[4771]: I1011 10:38:22.293760 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:38:22.294517 master-1 kubenswrapper[4771]: I1011 10:38:22.294463 4771 patch_prober.go:28] interesting pod/kube-controller-manager-guard-master-1 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" start-of-body= Oct 11 10:38:22.294622 master-1 kubenswrapper[4771]: I1011 10:38:22.294546 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podUID="a706deec-9223-4663-9db5-71147d242c34" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" Oct 11 10:38:22.447976 master-1 kubenswrapper[4771]: I1011 10:38:22.447748 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7c05954-4353-4cd1-9130-7fcb832a0493" path="/var/lib/kubelet/pods/a7c05954-4353-4cd1-9130-7fcb832a0493/volumes" Oct 11 10:38:22.452592 master-1 kubenswrapper[4771]: I1011 10:38:22.452429 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmzlm\" (UniqueName: \"kubernetes.io/projected/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-kube-api-access-mmzlm\") pod \"console-775ff6c4fc-csp4z\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:22.579567 master-1 kubenswrapper[4771]: I1011 10:38:22.579456 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:22.623839 master-1 kubenswrapper[4771]: I1011 10:38:22.623753 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-4-master-1" event={"ID":"66232733-dcfb-4320-a372-ce05d7d777d9","Type":"ContainerStarted","Data":"1a8de711412c9754f899398f555a2bb9a02c8065248c232ba0c054fb5b00ec21"} Oct 11 10:38:23.887668 master-1 kubenswrapper[4771]: I1011 10:38:23.887569 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-65bb9777fc-66jxg" Oct 11 10:38:24.021620 master-1 kubenswrapper[4771]: W1011 10:38:24.021530 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7aa64b_afab_4b3a_b56d_81c324e7a8cb.slice/crio-dbb5133e318020821233bd4743645ca9f974f8d4348733f58f43c17203dfa102 WatchSource:0}: Error finding container dbb5133e318020821233bd4743645ca9f974f8d4348733f58f43c17203dfa102: Status 404 returned error can't find the container with id dbb5133e318020821233bd4743645ca9f974f8d4348733f58f43c17203dfa102 Oct 11 10:38:24.126422 master-1 kubenswrapper[4771]: I1011 10:38:24.125878 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-775ff6c4fc-csp4z"] Oct 11 10:38:24.640607 master-1 kubenswrapper[4771]: I1011 10:38:24.640509 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775ff6c4fc-csp4z" event={"ID":"de7aa64b-afab-4b3a-b56d-81c324e7a8cb","Type":"ContainerStarted","Data":"10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5"} Oct 11 10:38:24.640888 master-1 kubenswrapper[4771]: I1011 10:38:24.640644 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775ff6c4fc-csp4z" event={"ID":"de7aa64b-afab-4b3a-b56d-81c324e7a8cb","Type":"ContainerStarted","Data":"dbb5133e318020821233bd4743645ca9f974f8d4348733f58f43c17203dfa102"} Oct 11 10:38:25.436821 master-1 kubenswrapper[4771]: I1011 10:38:25.436724 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:25.456819 master-1 kubenswrapper[4771]: I1011 10:38:25.456755 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="8e33f2eb-84da-43b0-9aad-9b9ee4940b82" Oct 11 10:38:25.456819 master-1 kubenswrapper[4771]: I1011 10:38:25.456803 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="8e33f2eb-84da-43b0-9aad-9b9ee4940b82" Oct 11 10:38:25.753541 master-1 kubenswrapper[4771]: I1011 10:38:25.753332 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/installer-4-master-1" podStartSLOduration=7.7532997 podStartE2EDuration="7.7532997s" podCreationTimestamp="2025-10-11 10:38:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:38:24.410873153 +0000 UTC m=+736.385099654" watchObservedRunningTime="2025-10-11 10:38:25.7532997 +0000 UTC m=+737.727526171" Oct 11 10:38:26.735630 master-1 kubenswrapper[4771]: I1011 10:38:26.735551 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:38:27.294651 master-1 kubenswrapper[4771]: I1011 10:38:27.294562 4771 patch_prober.go:28] interesting pod/kube-controller-manager-guard-master-1 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" start-of-body= Oct 11 10:38:27.294944 master-1 kubenswrapper[4771]: I1011 10:38:27.294656 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podUID="a706deec-9223-4663-9db5-71147d242c34" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" Oct 11 10:38:28.046218 master-1 kubenswrapper[4771]: I1011 10:38:28.045891 4771 kubelet.go:1914] "Deleted mirror pod because it is outdated" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:28.142590 master-1 kubenswrapper[4771]: I1011 10:38:28.142507 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:38:29.094589 master-1 kubenswrapper[4771]: I1011 10:38:29.093630 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-775ff6c4fc-csp4z" podStartSLOduration=40.093601824 podStartE2EDuration="40.093601824s" podCreationTimestamp="2025-10-11 10:37:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:38:29.092806081 +0000 UTC m=+741.067032592" watchObservedRunningTime="2025-10-11 10:38:29.093601824 +0000 UTC m=+741.067828305" Oct 11 10:38:30.645293 master-1 kubenswrapper[4771]: I1011 10:38:30.645209 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:30.647671 master-1 kubenswrapper[4771]: I1011 10:38:30.647605 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:38:30.680243 master-1 kubenswrapper[4771]: I1011 10:38:30.680156 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0b5de9d609ee1e6c379f71934cb2c3c6","Type":"ContainerStarted","Data":"4247c914a32e821feeb321db49e7b5b061a40ecb112a752686b9ea07098f462f"} Oct 11 10:38:31.689604 master-1 kubenswrapper[4771]: I1011 10:38:31.689519 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0b5de9d609ee1e6c379f71934cb2c3c6","Type":"ContainerStarted","Data":"9e6a4086932c3b4c0590b1992411e46984c974a11450de3378bede5ca3045d02"} Oct 11 10:38:32.298476 master-1 kubenswrapper[4771]: I1011 10:38:32.298426 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:38:32.580404 master-1 kubenswrapper[4771]: I1011 10:38:32.580298 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:32.580758 master-1 kubenswrapper[4771]: I1011 10:38:32.580431 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:38:32.582343 master-1 kubenswrapper[4771]: I1011 10:38:32.582287 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:38:32.582481 master-1 kubenswrapper[4771]: I1011 10:38:32.582427 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:38:32.697344 master-1 kubenswrapper[4771]: I1011 10:38:32.697243 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0b5de9d609ee1e6c379f71934cb2c3c6","Type":"ContainerStarted","Data":"913e0c188082961ad93b5f6a07d9eda57e62160ccbff129947e77948c758035a"} Oct 11 10:38:33.713722 master-1 kubenswrapper[4771]: I1011 10:38:33.713602 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0b5de9d609ee1e6c379f71934cb2c3c6","Type":"ContainerStarted","Data":"79e52bbf7393881dfbba04f7a9f71721266d98f1191a6c7be91f8bc0ce4e1139"} Oct 11 10:38:34.725504 master-1 kubenswrapper[4771]: I1011 10:38:34.725409 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"0b5de9d609ee1e6c379f71934cb2c3c6","Type":"ContainerStarted","Data":"068b46162b2804f4e661290cc4e58111faa3ee64a5ff733b8a30de9f4b7d070e"} Oct 11 10:38:35.011075 master-1 kubenswrapper[4771]: I1011 10:38:35.010878 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-apiserver/installer-4-master-1"] Oct 11 10:38:35.012957 master-1 kubenswrapper[4771]: I1011 10:38:35.011351 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/installer-4-master-1" podUID="66232733-dcfb-4320-a372-ce05d7d777d9" containerName="installer" containerID="cri-o://1a8de711412c9754f899398f555a2bb9a02c8065248c232ba0c054fb5b00ec21" gracePeriod=30 Oct 11 10:38:35.033141 master-1 kubenswrapper[4771]: I1011 10:38:35.033036 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podStartSLOduration=6.033009036 podStartE2EDuration="6.033009036s" podCreationTimestamp="2025-10-11 10:38:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:38:35.031167853 +0000 UTC m=+747.005394314" watchObservedRunningTime="2025-10-11 10:38:35.033009036 +0000 UTC m=+747.007235477" Oct 11 10:38:40.646632 master-1 kubenswrapper[4771]: I1011 10:38:40.646451 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:40.647695 master-1 kubenswrapper[4771]: I1011 10:38:40.646668 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:40.647695 master-1 kubenswrapper[4771]: I1011 10:38:40.646700 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:40.647695 master-1 kubenswrapper[4771]: I1011 10:38:40.646723 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:40.653845 master-1 kubenswrapper[4771]: I1011 10:38:40.653783 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:40.654103 master-1 kubenswrapper[4771]: I1011 10:38:40.654047 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:41.765591 master-1 kubenswrapper[4771]: I1011 10:38:41.765528 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/installer-5-master-1"] Oct 11 10:38:41.766610 master-1 kubenswrapper[4771]: I1011 10:38:41.766407 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:41.792963 master-1 kubenswrapper[4771]: I1011 10:38:41.792885 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:41.828765 master-1 kubenswrapper[4771]: I1011 10:38:41.828628 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-var-lock\") pod \"installer-5-master-1\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:41.828765 master-1 kubenswrapper[4771]: I1011 10:38:41.828769 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:41.829334 master-1 kubenswrapper[4771]: I1011 10:38:41.828843 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kube-api-access\") pod \"installer-5-master-1\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:41.929881 master-1 kubenswrapper[4771]: I1011 10:38:41.929787 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:41.929881 master-1 kubenswrapper[4771]: I1011 10:38:41.929882 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kube-api-access\") pod \"installer-5-master-1\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:41.930321 master-1 kubenswrapper[4771]: I1011 10:38:41.929924 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-var-lock\") pod \"installer-5-master-1\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:41.930321 master-1 kubenswrapper[4771]: I1011 10:38:41.929980 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kubelet-dir\") pod \"installer-5-master-1\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:41.930321 master-1 kubenswrapper[4771]: I1011 10:38:41.929999 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-var-lock\") pod \"installer-5-master-1\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:42.324612 master-1 kubenswrapper[4771]: I1011 10:38:42.324540 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-5-master-1"] Oct 11 10:38:42.581791 master-1 kubenswrapper[4771]: I1011 10:38:42.581584 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:38:42.581791 master-1 kubenswrapper[4771]: I1011 10:38:42.581687 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:38:42.695065 master-1 kubenswrapper[4771]: I1011 10:38:42.694814 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kube-api-access\") pod \"installer-5-master-1\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:42.696469 master-1 kubenswrapper[4771]: I1011 10:38:42.695632 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:38:42.806976 master-1 kubenswrapper[4771]: I1011 10:38:42.806860 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:38:43.172895 master-1 kubenswrapper[4771]: I1011 10:38:43.171549 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-5-master-1"] Oct 11 10:38:43.178656 master-1 kubenswrapper[4771]: W1011 10:38:43.178578 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04d0b40e_b6ae_4466_a0af_fcb5ce630a97.slice/crio-bbc3a84466c188ab0111b2748b205176a81652b14bfe38d9a3a683ab12c3236b WatchSource:0}: Error finding container bbc3a84466c188ab0111b2748b205176a81652b14bfe38d9a3a683ab12c3236b: Status 404 returned error can't find the container with id bbc3a84466c188ab0111b2748b205176a81652b14bfe38d9a3a683ab12c3236b Oct 11 10:38:43.803014 master-1 kubenswrapper[4771]: I1011 10:38:43.802954 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-5-master-1" event={"ID":"04d0b40e-b6ae-4466-a0af-fcb5ce630a97","Type":"ContainerStarted","Data":"3d3a7650ee6f21f1edc22785fe9fc463251f973399b34912c74a0d533d0b5e22"} Oct 11 10:38:43.803545 master-1 kubenswrapper[4771]: I1011 10:38:43.803506 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-5-master-1" event={"ID":"04d0b40e-b6ae-4466-a0af-fcb5ce630a97","Type":"ContainerStarted","Data":"bbc3a84466c188ab0111b2748b205176a81652b14bfe38d9a3a683ab12c3236b"} Oct 11 10:38:43.830700 master-1 kubenswrapper[4771]: I1011 10:38:43.830601 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/installer-5-master-1" podStartSLOduration=2.830584633 podStartE2EDuration="2.830584633s" podCreationTimestamp="2025-10-11 10:38:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:38:43.824179597 +0000 UTC m=+755.798406048" watchObservedRunningTime="2025-10-11 10:38:43.830584633 +0000 UTC m=+755.804811074" Oct 11 10:38:52.581221 master-1 kubenswrapper[4771]: I1011 10:38:52.581105 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:38:52.581221 master-1 kubenswrapper[4771]: I1011 10:38:52.581213 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:38:54.884729 master-1 kubenswrapper[4771]: I1011 10:38:54.884645 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_installer-4-master-1_66232733-dcfb-4320-a372-ce05d7d777d9/installer/0.log" Oct 11 10:38:54.885549 master-1 kubenswrapper[4771]: I1011 10:38:54.884731 4771 generic.go:334] "Generic (PLEG): container finished" podID="66232733-dcfb-4320-a372-ce05d7d777d9" containerID="1a8de711412c9754f899398f555a2bb9a02c8065248c232ba0c054fb5b00ec21" exitCode=1 Oct 11 10:38:54.885549 master-1 kubenswrapper[4771]: I1011 10:38:54.884775 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-4-master-1" event={"ID":"66232733-dcfb-4320-a372-ce05d7d777d9","Type":"ContainerDied","Data":"1a8de711412c9754f899398f555a2bb9a02c8065248c232ba0c054fb5b00ec21"} Oct 11 10:38:55.264117 master-1 kubenswrapper[4771]: I1011 10:38:55.264067 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_installer-4-master-1_66232733-dcfb-4320-a372-ce05d7d777d9/installer/0.log" Oct 11 10:38:55.264490 master-1 kubenswrapper[4771]: I1011 10:38:55.264148 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:55.377803 master-1 kubenswrapper[4771]: I1011 10:38:55.377626 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-var-lock\") pod \"66232733-dcfb-4320-a372-ce05d7d777d9\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " Oct 11 10:38:55.378068 master-1 kubenswrapper[4771]: I1011 10:38:55.377789 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-kubelet-dir\") pod \"66232733-dcfb-4320-a372-ce05d7d777d9\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " Oct 11 10:38:55.378463 master-1 kubenswrapper[4771]: I1011 10:38:55.377894 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/66232733-dcfb-4320-a372-ce05d7d777d9-kube-api-access\") pod \"66232733-dcfb-4320-a372-ce05d7d777d9\" (UID: \"66232733-dcfb-4320-a372-ce05d7d777d9\") " Oct 11 10:38:55.378696 master-1 kubenswrapper[4771]: I1011 10:38:55.377727 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-var-lock" (OuterVolumeSpecName: "var-lock") pod "66232733-dcfb-4320-a372-ce05d7d777d9" (UID: "66232733-dcfb-4320-a372-ce05d7d777d9"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:38:55.378848 master-1 kubenswrapper[4771]: I1011 10:38:55.377818 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "66232733-dcfb-4320-a372-ce05d7d777d9" (UID: "66232733-dcfb-4320-a372-ce05d7d777d9"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:38:55.378969 master-1 kubenswrapper[4771]: I1011 10:38:55.378918 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:55.378969 master-1 kubenswrapper[4771]: I1011 10:38:55.378956 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66232733-dcfb-4320-a372-ce05d7d777d9-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:55.381448 master-1 kubenswrapper[4771]: I1011 10:38:55.381403 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66232733-dcfb-4320-a372-ce05d7d777d9-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "66232733-dcfb-4320-a372-ce05d7d777d9" (UID: "66232733-dcfb-4320-a372-ce05d7d777d9"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:38:55.479832 master-1 kubenswrapper[4771]: I1011 10:38:55.479748 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/66232733-dcfb-4320-a372-ce05d7d777d9-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:38:55.901024 master-1 kubenswrapper[4771]: I1011 10:38:55.900942 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_installer-4-master-1_66232733-dcfb-4320-a372-ce05d7d777d9/installer/0.log" Oct 11 10:38:55.901695 master-1 kubenswrapper[4771]: I1011 10:38:55.901049 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-4-master-1" event={"ID":"66232733-dcfb-4320-a372-ce05d7d777d9","Type":"ContainerDied","Data":"210c1e83cdd57074a16669f3f9ab89020bba5fd50626a163674b365ff40935aa"} Oct 11 10:38:55.901695 master-1 kubenswrapper[4771]: I1011 10:38:55.901113 4771 scope.go:117] "RemoveContainer" containerID="1a8de711412c9754f899398f555a2bb9a02c8065248c232ba0c054fb5b00ec21" Oct 11 10:38:55.901695 master-1 kubenswrapper[4771]: I1011 10:38:55.901329 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-4-master-1" Oct 11 10:38:55.952080 master-1 kubenswrapper[4771]: I1011 10:38:55.951993 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-apiserver/installer-4-master-1"] Oct 11 10:38:55.958227 master-1 kubenswrapper[4771]: I1011 10:38:55.958130 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-apiserver/installer-4-master-1"] Oct 11 10:38:56.447294 master-1 kubenswrapper[4771]: I1011 10:38:56.447185 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66232733-dcfb-4320-a372-ce05d7d777d9" path="/var/lib/kubelet/pods/66232733-dcfb-4320-a372-ce05d7d777d9/volumes" Oct 11 10:39:00.615250 master-1 kubenswrapper[4771]: I1011 10:39:00.615190 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-kntdb"] Oct 11 10:39:00.616297 master-1 kubenswrapper[4771]: E1011 10:39:00.615476 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="66232733-dcfb-4320-a372-ce05d7d777d9" containerName="installer" Oct 11 10:39:00.616297 master-1 kubenswrapper[4771]: I1011 10:39:00.615510 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="66232733-dcfb-4320-a372-ce05d7d777d9" containerName="installer" Oct 11 10:39:00.616297 master-1 kubenswrapper[4771]: I1011 10:39:00.615652 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="66232733-dcfb-4320-a372-ce05d7d777d9" containerName="installer" Oct 11 10:39:00.616297 master-1 kubenswrapper[4771]: I1011 10:39:00.616246 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.619579 master-1 kubenswrapper[4771]: I1011 10:39:00.619524 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-image-registry"/"image-registry-certificates" Oct 11 10:39:00.619660 master-1 kubenswrapper[4771]: I1011 10:39:00.619531 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-image-registry"/"node-ca-dockercfg-cs6lc" Oct 11 10:39:00.619948 master-1 kubenswrapper[4771]: I1011 10:39:00.619895 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-image-registry"/"kube-root-ca.crt" Oct 11 10:39:00.621658 master-1 kubenswrapper[4771]: I1011 10:39:00.621585 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-image-registry"/"openshift-service-ca.crt" Oct 11 10:39:00.761041 master-1 kubenswrapper[4771]: I1011 10:39:00.760963 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6f2m\" (UniqueName: \"kubernetes.io/projected/f621f971-6560-4be2-b36c-307a440c0769-kube-api-access-z6f2m\") pod \"node-ca-kntdb\" (UID: \"f621f971-6560-4be2-b36c-307a440c0769\") " pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.761465 master-1 kubenswrapper[4771]: I1011 10:39:00.761395 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f621f971-6560-4be2-b36c-307a440c0769-host\") pod \"node-ca-kntdb\" (UID: \"f621f971-6560-4be2-b36c-307a440c0769\") " pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.761554 master-1 kubenswrapper[4771]: I1011 10:39:00.761514 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f621f971-6560-4be2-b36c-307a440c0769-serviceca\") pod \"node-ca-kntdb\" (UID: \"f621f971-6560-4be2-b36c-307a440c0769\") " pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.863265 master-1 kubenswrapper[4771]: I1011 10:39:00.863187 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f621f971-6560-4be2-b36c-307a440c0769-host\") pod \"node-ca-kntdb\" (UID: \"f621f971-6560-4be2-b36c-307a440c0769\") " pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.863500 master-1 kubenswrapper[4771]: I1011 10:39:00.863311 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f621f971-6560-4be2-b36c-307a440c0769-serviceca\") pod \"node-ca-kntdb\" (UID: \"f621f971-6560-4be2-b36c-307a440c0769\") " pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.863500 master-1 kubenswrapper[4771]: I1011 10:39:00.863383 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f621f971-6560-4be2-b36c-307a440c0769-host\") pod \"node-ca-kntdb\" (UID: \"f621f971-6560-4be2-b36c-307a440c0769\") " pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.864447 master-1 kubenswrapper[4771]: I1011 10:39:00.864403 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f621f971-6560-4be2-b36c-307a440c0769-serviceca\") pod \"node-ca-kntdb\" (UID: \"f621f971-6560-4be2-b36c-307a440c0769\") " pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.864717 master-1 kubenswrapper[4771]: I1011 10:39:00.864682 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-z6f2m\" (UniqueName: \"kubernetes.io/projected/f621f971-6560-4be2-b36c-307a440c0769-kube-api-access-z6f2m\") pod \"node-ca-kntdb\" (UID: \"f621f971-6560-4be2-b36c-307a440c0769\") " pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.890656 master-1 kubenswrapper[4771]: I1011 10:39:00.890560 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6f2m\" (UniqueName: \"kubernetes.io/projected/f621f971-6560-4be2-b36c-307a440c0769-kube-api-access-z6f2m\") pod \"node-ca-kntdb\" (UID: \"f621f971-6560-4be2-b36c-307a440c0769\") " pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.935236 master-1 kubenswrapper[4771]: I1011 10:39:00.935165 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kntdb" Oct 11 10:39:00.950457 master-1 kubenswrapper[4771]: W1011 10:39:00.950410 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf621f971_6560_4be2_b36c_307a440c0769.slice/crio-180bb5a314b1530c0f87a385216eb06130a4145266776e64ae7491dd6e872065 WatchSource:0}: Error finding container 180bb5a314b1530c0f87a385216eb06130a4145266776e64ae7491dd6e872065: Status 404 returned error can't find the container with id 180bb5a314b1530c0f87a385216eb06130a4145266776e64ae7491dd6e872065 Oct 11 10:39:01.475053 master-1 kubenswrapper[4771]: I1011 10:39:01.474961 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:39:01.475307 master-1 kubenswrapper[4771]: E1011 10:39:01.475169 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker podName:d7647696-42d9-4dd9-bc3b-a4d52a42cf9a nodeName:}" failed. No retries permitted until 2025-10-11 10:41:03.475142578 +0000 UTC m=+895.449369029 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker") pod "operator-controller-controller-manager-668cb7cdc8-bqdlc" (UID: "d7647696-42d9-4dd9-bc3b-a4d52a42cf9a") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:39:01.576818 master-1 kubenswrapper[4771]: I1011 10:39:01.576714 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:39:01.577081 master-1 kubenswrapper[4771]: E1011 10:39:01.576929 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker podName:6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b nodeName:}" failed. No retries permitted until 2025-10-11 10:41:03.576905376 +0000 UTC m=+895.551131817 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "etc-docker" (UniqueName: "kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker") pod "catalogd-controller-manager-596f9d8bbf-tpzsm" (UID: "6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b") : hostPath type check failed: /etc/docker is not a directory Oct 11 10:39:01.947485 master-1 kubenswrapper[4771]: I1011 10:39:01.947415 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kntdb" event={"ID":"f621f971-6560-4be2-b36c-307a440c0769","Type":"ContainerStarted","Data":"180bb5a314b1530c0f87a385216eb06130a4145266776e64ae7491dd6e872065"} Oct 11 10:39:02.580133 master-1 kubenswrapper[4771]: I1011 10:39:02.580067 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:39:02.580466 master-1 kubenswrapper[4771]: I1011 10:39:02.580139 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:39:03.894939 master-1 kubenswrapper[4771]: E1011 10:39:03.894851 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" podUID="d7647696-42d9-4dd9-bc3b-a4d52a42cf9a" Oct 11 10:39:03.895803 master-1 kubenswrapper[4771]: E1011 10:39:03.895181 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[etc-docker], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" podUID="6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b" Oct 11 10:39:03.963642 master-1 kubenswrapper[4771]: I1011 10:39:03.963546 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kntdb" event={"ID":"f621f971-6560-4be2-b36c-307a440c0769","Type":"ContainerStarted","Data":"d6296cb88992a02f56ed761cb9e4574f8959587dafa90daed9fe2b15ced3e3a0"} Oct 11 10:39:03.963642 master-1 kubenswrapper[4771]: I1011 10:39:03.963590 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:39:03.963918 master-1 kubenswrapper[4771]: I1011 10:39:03.963715 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:39:12.582753 master-1 kubenswrapper[4771]: I1011 10:39:12.582656 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:39:12.583970 master-1 kubenswrapper[4771]: I1011 10:39:12.582805 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:39:22.580769 master-1 kubenswrapper[4771]: I1011 10:39:22.580695 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:39:22.581798 master-1 kubenswrapper[4771]: I1011 10:39:22.580806 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:39:32.581540 master-1 kubenswrapper[4771]: I1011 10:39:32.581421 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:39:32.581540 master-1 kubenswrapper[4771]: I1011 10:39:32.581521 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:39:42.480877 master-1 kubenswrapper[4771]: I1011 10:39:42.480753 4771 kubelet.go:2431] "SyncLoop REMOVE" source="file" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:39:42.482178 master-1 kubenswrapper[4771]: I1011 10:39:42.481180 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver" containerID="cri-o://be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a" gracePeriod=135 Oct 11 10:39:42.482178 master-1 kubenswrapper[4771]: I1011 10:39:42.481265 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-check-endpoints" containerID="cri-o://50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07" gracePeriod=135 Oct 11 10:39:42.482178 master-1 kubenswrapper[4771]: I1011 10:39:42.481384 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-cert-regeneration-controller" containerID="cri-o://7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109" gracePeriod=135 Oct 11 10:39:42.482178 master-1 kubenswrapper[4771]: I1011 10:39:42.481464 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-insecure-readyz" containerID="cri-o://637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4" gracePeriod=135 Oct 11 10:39:42.482178 master-1 kubenswrapper[4771]: I1011 10:39:42.481535 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-cert-syncer" containerID="cri-o://0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd" gracePeriod=135 Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: I1011 10:39:42.483282 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: E1011 10:39:42.483732 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-cert-syncer" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: I1011 10:39:42.483765 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-cert-syncer" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: E1011 10:39:42.483793 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="setup" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: I1011 10:39:42.483813 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="setup" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: E1011 10:39:42.483846 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: I1011 10:39:42.483864 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: E1011 10:39:42.483892 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-cert-regeneration-controller" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: I1011 10:39:42.483911 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-cert-regeneration-controller" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: E1011 10:39:42.483937 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-check-endpoints" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: I1011 10:39:42.483955 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-check-endpoints" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: E1011 10:39:42.483978 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-insecure-readyz" Oct 11 10:39:42.483985 master-1 kubenswrapper[4771]: I1011 10:39:42.483992 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-insecure-readyz" Oct 11 10:39:42.485029 master-1 kubenswrapper[4771]: I1011 10:39:42.484189 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-cert-regeneration-controller" Oct 11 10:39:42.485029 master-1 kubenswrapper[4771]: I1011 10:39:42.484213 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-cert-syncer" Oct 11 10:39:42.485029 master-1 kubenswrapper[4771]: I1011 10:39:42.484230 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-insecure-readyz" Oct 11 10:39:42.485029 master-1 kubenswrapper[4771]: I1011 10:39:42.484254 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver" Oct 11 10:39:42.485029 master-1 kubenswrapper[4771]: I1011 10:39:42.484268 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver-check-endpoints" Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: I1011 10:39:42.514959 4771 patch_prober.go:28] interesting pod/kube-apiserver-master-1 container/kube-apiserver namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:39:42.515025 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:39:42.517506 master-1 kubenswrapper[4771]: I1011 10:39:42.515033 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="e39186c2ebd02622803bdbec6984de2a" containerName="kube-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:39:42.581170 master-1 kubenswrapper[4771]: I1011 10:39:42.581064 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:39:42.581407 master-1 kubenswrapper[4771]: I1011 10:39:42.581182 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:39:42.591940 master-1 kubenswrapper[4771]: I1011 10:39:42.591865 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:39:42.592101 master-1 kubenswrapper[4771]: I1011 10:39:42.591985 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:39:42.592101 master-1 kubenswrapper[4771]: I1011 10:39:42.592058 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:39:42.694068 master-1 kubenswrapper[4771]: I1011 10:39:42.693595 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:39:42.694417 master-1 kubenswrapper[4771]: I1011 10:39:42.693982 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:39:42.694417 master-1 kubenswrapper[4771]: I1011 10:39:42.694173 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:39:42.694417 master-1 kubenswrapper[4771]: I1011 10:39:42.694091 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:39:42.694631 master-1 kubenswrapper[4771]: I1011 10:39:42.694415 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:39:42.694715 master-1 kubenswrapper[4771]: I1011 10:39:42.694617 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:39:43.225377 master-1 kubenswrapper[4771]: I1011 10:39:43.225289 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_e39186c2ebd02622803bdbec6984de2a/kube-apiserver-cert-syncer/0.log" Oct 11 10:39:43.226057 master-1 kubenswrapper[4771]: I1011 10:39:43.226019 4771 generic.go:334] "Generic (PLEG): container finished" podID="e39186c2ebd02622803bdbec6984de2a" containerID="50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07" exitCode=0 Oct 11 10:39:43.226057 master-1 kubenswrapper[4771]: I1011 10:39:43.226054 4771 generic.go:334] "Generic (PLEG): container finished" podID="e39186c2ebd02622803bdbec6984de2a" containerID="637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4" exitCode=0 Oct 11 10:39:43.226147 master-1 kubenswrapper[4771]: I1011 10:39:43.226065 4771 generic.go:334] "Generic (PLEG): container finished" podID="e39186c2ebd02622803bdbec6984de2a" containerID="7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109" exitCode=0 Oct 11 10:39:43.226147 master-1 kubenswrapper[4771]: I1011 10:39:43.226077 4771 generic.go:334] "Generic (PLEG): container finished" podID="e39186c2ebd02622803bdbec6984de2a" containerID="0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd" exitCode=2 Oct 11 10:39:43.227632 master-1 kubenswrapper[4771]: I1011 10:39:43.227563 4771 generic.go:334] "Generic (PLEG): container finished" podID="04d0b40e-b6ae-4466-a0af-fcb5ce630a97" containerID="3d3a7650ee6f21f1edc22785fe9fc463251f973399b34912c74a0d533d0b5e22" exitCode=0 Oct 11 10:39:43.227710 master-1 kubenswrapper[4771]: I1011 10:39:43.227667 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-5-master-1" event={"ID":"04d0b40e-b6ae-4466-a0af-fcb5ce630a97","Type":"ContainerDied","Data":"3d3a7650ee6f21f1edc22785fe9fc463251f973399b34912c74a0d533d0b5e22"} Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: I1011 10:39:43.241722 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:39:43.241805 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:39:43.243171 master-1 kubenswrapper[4771]: I1011 10:39:43.241820 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:39:43.384072 master-1 kubenswrapper[4771]: I1011 10:39:43.383999 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="e39186c2ebd02622803bdbec6984de2a" podUID="42d61efaa0f96869cf2939026aad6022" Oct 11 10:39:43.485838 master-1 kubenswrapper[4771]: I1011 10:39:43.485648 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kntdb" podStartSLOduration=52.448594775 podStartE2EDuration="54.485623726s" podCreationTimestamp="2025-10-11 10:38:49 +0000 UTC" firstStartedPulling="2025-10-11 10:39:00.952758982 +0000 UTC m=+772.926985453" lastFinishedPulling="2025-10-11 10:39:02.989787953 +0000 UTC m=+774.964014404" observedRunningTime="2025-10-11 10:39:04.188439094 +0000 UTC m=+776.162665595" watchObservedRunningTime="2025-10-11 10:39:43.485623726 +0000 UTC m=+815.459850207" Oct 11 10:39:44.544845 master-1 kubenswrapper[4771]: I1011 10:39:44.544770 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:39:44.722198 master-1 kubenswrapper[4771]: I1011 10:39:44.722108 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-var-lock\") pod \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " Oct 11 10:39:44.722198 master-1 kubenswrapper[4771]: I1011 10:39:44.722203 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kube-api-access\") pod \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " Oct 11 10:39:44.722612 master-1 kubenswrapper[4771]: I1011 10:39:44.722279 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kubelet-dir\") pod \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\" (UID: \"04d0b40e-b6ae-4466-a0af-fcb5ce630a97\") " Oct 11 10:39:44.723194 master-1 kubenswrapper[4771]: I1011 10:39:44.723019 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "04d0b40e-b6ae-4466-a0af-fcb5ce630a97" (UID: "04d0b40e-b6ae-4466-a0af-fcb5ce630a97"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:39:44.723194 master-1 kubenswrapper[4771]: I1011 10:39:44.723085 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-var-lock" (OuterVolumeSpecName: "var-lock") pod "04d0b40e-b6ae-4466-a0af-fcb5ce630a97" (UID: "04d0b40e-b6ae-4466-a0af-fcb5ce630a97"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:39:44.728465 master-1 kubenswrapper[4771]: I1011 10:39:44.728407 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "04d0b40e-b6ae-4466-a0af-fcb5ce630a97" (UID: "04d0b40e-b6ae-4466-a0af-fcb5ce630a97"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:39:44.824508 master-1 kubenswrapper[4771]: I1011 10:39:44.824439 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:39:44.824508 master-1 kubenswrapper[4771]: I1011 10:39:44.824483 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:39:44.824508 master-1 kubenswrapper[4771]: I1011 10:39:44.824495 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04d0b40e-b6ae-4466-a0af-fcb5ce630a97-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:39:45.242203 master-1 kubenswrapper[4771]: I1011 10:39:45.242120 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-5-master-1" event={"ID":"04d0b40e-b6ae-4466-a0af-fcb5ce630a97","Type":"ContainerDied","Data":"bbc3a84466c188ab0111b2748b205176a81652b14bfe38d9a3a683ab12c3236b"} Oct 11 10:39:45.242203 master-1 kubenswrapper[4771]: I1011 10:39:45.242180 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbc3a84466c188ab0111b2748b205176a81652b14bfe38d9a3a683ab12c3236b" Oct 11 10:39:45.242597 master-1 kubenswrapper[4771]: I1011 10:39:45.242261 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-5-master-1" Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: I1011 10:39:48.242605 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:39:48.242791 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:39:48.244532 master-1 kubenswrapper[4771]: I1011 10:39:48.242779 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:39:52.580685 master-1 kubenswrapper[4771]: I1011 10:39:52.580474 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:39:52.580685 master-1 kubenswrapper[4771]: I1011 10:39:52.580623 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: I1011 10:39:53.243227 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:39:53.243300 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:39:53.244635 master-1 kubenswrapper[4771]: I1011 10:39:53.243321 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:39:53.244635 master-1 kubenswrapper[4771]: I1011 10:39:53.243452 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: I1011 10:39:53.247566 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:39:53.247615 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:39:53.248686 master-1 kubenswrapper[4771]: I1011 10:39:53.247624 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:39:56.658558 master-1 kubenswrapper[4771]: I1011 10:39:56.658450 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-scheduler/revision-pruner-6-master-1"] Oct 11 10:39:56.659925 master-1 kubenswrapper[4771]: E1011 10:39:56.658847 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="04d0b40e-b6ae-4466-a0af-fcb5ce630a97" containerName="installer" Oct 11 10:39:56.659925 master-1 kubenswrapper[4771]: I1011 10:39:56.658875 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d0b40e-b6ae-4466-a0af-fcb5ce630a97" containerName="installer" Oct 11 10:39:56.659925 master-1 kubenswrapper[4771]: I1011 10:39:56.659043 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="04d0b40e-b6ae-4466-a0af-fcb5ce630a97" containerName="installer" Oct 11 10:39:56.660239 master-1 kubenswrapper[4771]: I1011 10:39:56.659994 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:39:56.662948 master-1 kubenswrapper[4771]: I1011 10:39:56.662878 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-kube-scheduler"/"installer-sa-dockercfg-js756" Oct 11 10:39:56.670625 master-1 kubenswrapper[4771]: I1011 10:39:56.670348 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/revision-pruner-6-master-1"] Oct 11 10:39:56.723719 master-1 kubenswrapper[4771]: I1011 10:39:56.723633 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kube-api-access\") pod \"revision-pruner-6-master-1\" (UID: \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\") " pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:39:56.724154 master-1 kubenswrapper[4771]: I1011 10:39:56.724116 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kubelet-dir\") pod \"revision-pruner-6-master-1\" (UID: \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\") " pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:39:56.826721 master-1 kubenswrapper[4771]: I1011 10:39:56.826587 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kube-api-access\") pod \"revision-pruner-6-master-1\" (UID: \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\") " pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:39:56.827299 master-1 kubenswrapper[4771]: I1011 10:39:56.827263 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kubelet-dir\") pod \"revision-pruner-6-master-1\" (UID: \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\") " pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:39:56.827699 master-1 kubenswrapper[4771]: I1011 10:39:56.827472 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kubelet-dir\") pod \"revision-pruner-6-master-1\" (UID: \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\") " pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:39:56.848295 master-1 kubenswrapper[4771]: I1011 10:39:56.848229 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kube-api-access\") pod \"revision-pruner-6-master-1\" (UID: \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\") " pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:39:56.984570 master-1 kubenswrapper[4771]: I1011 10:39:56.984311 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:39:57.454949 master-1 kubenswrapper[4771]: I1011 10:39:57.454544 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-scheduler/revision-pruner-6-master-1"] Oct 11 10:39:57.468809 master-1 kubenswrapper[4771]: W1011 10:39:57.468739 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d28ccd7_4f37_4a24_a9ff_ef97ff08ae05.slice/crio-713d4071ed7a38a97107ad5f1629e72b0b8fd55934c7595a21204f0f11e35c02 WatchSource:0}: Error finding container 713d4071ed7a38a97107ad5f1629e72b0b8fd55934c7595a21204f0f11e35c02: Status 404 returned error can't find the container with id 713d4071ed7a38a97107ad5f1629e72b0b8fd55934c7595a21204f0f11e35c02 Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: I1011 10:39:58.245866 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:39:58.245986 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:39:58.250892 master-1 kubenswrapper[4771]: I1011 10:39:58.250841 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:39:58.319146 master-1 kubenswrapper[4771]: I1011 10:39:58.319079 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/revision-pruner-6-master-1" event={"ID":"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05","Type":"ContainerStarted","Data":"ec9f0592baf8c5dd0d5f3cecb9104d9b2a55e7f1b365b952640c85483ccbac69"} Oct 11 10:39:58.319146 master-1 kubenswrapper[4771]: I1011 10:39:58.319148 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/revision-pruner-6-master-1" event={"ID":"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05","Type":"ContainerStarted","Data":"713d4071ed7a38a97107ad5f1629e72b0b8fd55934c7595a21204f0f11e35c02"} Oct 11 10:39:58.769923 master-1 kubenswrapper[4771]: I1011 10:39:58.769790 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-scheduler/revision-pruner-6-master-1" podStartSLOduration=2.769763861 podStartE2EDuration="2.769763861s" podCreationTimestamp="2025-10-11 10:39:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:39:58.689148724 +0000 UTC m=+830.663375195" watchObservedRunningTime="2025-10-11 10:39:58.769763861 +0000 UTC m=+830.743990302" Oct 11 10:39:59.327900 master-1 kubenswrapper[4771]: I1011 10:39:59.327806 4771 generic.go:334] "Generic (PLEG): container finished" podID="3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05" containerID="ec9f0592baf8c5dd0d5f3cecb9104d9b2a55e7f1b365b952640c85483ccbac69" exitCode=0 Oct 11 10:39:59.327900 master-1 kubenswrapper[4771]: I1011 10:39:59.327872 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/revision-pruner-6-master-1" event={"ID":"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05","Type":"ContainerDied","Data":"ec9f0592baf8c5dd0d5f3cecb9104d9b2a55e7f1b365b952640c85483ccbac69"} Oct 11 10:40:00.797335 master-1 kubenswrapper[4771]: I1011 10:40:00.797280 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:40:00.890681 master-1 kubenswrapper[4771]: I1011 10:40:00.890581 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kube-api-access\") pod \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\" (UID: \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\") " Oct 11 10:40:00.890681 master-1 kubenswrapper[4771]: I1011 10:40:00.890675 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kubelet-dir\") pod \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\" (UID: \"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05\") " Oct 11 10:40:00.891288 master-1 kubenswrapper[4771]: I1011 10:40:00.890870 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05" (UID: "3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:40:00.907463 master-1 kubenswrapper[4771]: I1011 10:40:00.903815 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05" (UID: "3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:40:00.992867 master-1 kubenswrapper[4771]: I1011 10:40:00.992675 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:40:00.993276 master-1 kubenswrapper[4771]: I1011 10:40:00.993239 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:40:01.342154 master-1 kubenswrapper[4771]: I1011 10:40:01.342094 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-scheduler/revision-pruner-6-master-1" event={"ID":"3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05","Type":"ContainerDied","Data":"713d4071ed7a38a97107ad5f1629e72b0b8fd55934c7595a21204f0f11e35c02"} Oct 11 10:40:01.342563 master-1 kubenswrapper[4771]: I1011 10:40:01.342534 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="713d4071ed7a38a97107ad5f1629e72b0b8fd55934c7595a21204f0f11e35c02" Oct 11 10:40:01.342732 master-1 kubenswrapper[4771]: I1011 10:40:01.342229 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-scheduler/revision-pruner-6-master-1" Oct 11 10:40:02.581021 master-1 kubenswrapper[4771]: I1011 10:40:02.580941 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:40:02.583656 master-1 kubenswrapper[4771]: I1011 10:40:02.583611 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: I1011 10:40:03.245798 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:03.245991 master-1 kubenswrapper[4771]: I1011 10:40:03.245922 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:03.995851 master-1 kubenswrapper[4771]: I1011 10:40:03.995736 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-1"] Oct 11 10:40:03.996813 master-1 kubenswrapper[4771]: E1011 10:40:03.996560 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05" containerName="pruner" Oct 11 10:40:03.996813 master-1 kubenswrapper[4771]: I1011 10:40:03.996596 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05" containerName="pruner" Oct 11 10:40:03.997143 master-1 kubenswrapper[4771]: I1011 10:40:03.997094 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="3d28ccd7-4f37-4a24-a9ff-ef97ff08ae05" containerName="pruner" Oct 11 10:40:04.008077 master-1 kubenswrapper[4771]: I1011 10:40:04.007988 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.011635 master-1 kubenswrapper[4771]: I1011 10:40:04.011572 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"alertmanager-main-generated" Oct 11 10:40:04.012068 master-1 kubenswrapper[4771]: I1011 10:40:04.012042 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"alertmanager-kube-rbac-proxy-metric" Oct 11 10:40:04.012316 master-1 kubenswrapper[4771]: I1011 10:40:04.012264 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"alertmanager-kube-rbac-proxy-web" Oct 11 10:40:04.013039 master-1 kubenswrapper[4771]: I1011 10:40:04.012939 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"alertmanager-main-tls" Oct 11 10:40:04.013039 master-1 kubenswrapper[4771]: I1011 10:40:04.012977 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"alertmanager-main-tls-assets-0" Oct 11 10:40:04.013312 master-1 kubenswrapper[4771]: I1011 10:40:04.013071 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"alertmanager-main-web-config" Oct 11 10:40:04.013312 master-1 kubenswrapper[4771]: I1011 10:40:04.013111 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"alertmanager-kube-rbac-proxy" Oct 11 10:40:04.024535 master-1 kubenswrapper[4771]: I1011 10:40:04.024308 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"alertmanager-trusted-ca-bundle" Oct 11 10:40:04.033788 master-1 kubenswrapper[4771]: I1011 10:40:04.033703 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-1"] Oct 11 10:40:04.140334 master-1 kubenswrapper[4771]: I1011 10:40:04.140235 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8cmk\" (UniqueName: \"kubernetes.io/projected/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-kube-api-access-b8cmk\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.140334 master-1 kubenswrapper[4771]: I1011 10:40:04.140317 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-main-tls\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.140778 master-1 kubenswrapper[4771]: I1011 10:40:04.140424 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-metrics-client-ca\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.140778 master-1 kubenswrapper[4771]: I1011 10:40:04.140484 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.140778 master-1 kubenswrapper[4771]: I1011 10:40:04.140520 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-config-volume\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.140778 master-1 kubenswrapper[4771]: I1011 10:40:04.140644 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-tls-assets\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.140778 master-1 kubenswrapper[4771]: I1011 10:40:04.140703 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-web-config\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.140778 master-1 kubenswrapper[4771]: I1011 10:40:04.140773 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.141276 master-1 kubenswrapper[4771]: I1011 10:40:04.140848 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-config-out\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.141276 master-1 kubenswrapper[4771]: I1011 10:40:04.140884 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-alertmanager-main-db\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.141276 master-1 kubenswrapper[4771]: I1011 10:40:04.140917 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.141276 master-1 kubenswrapper[4771]: I1011 10:40:04.141001 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.242579 master-1 kubenswrapper[4771]: I1011 10:40:04.242516 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-config-out\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.242956 master-1 kubenswrapper[4771]: I1011 10:40:04.242925 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-alertmanager-main-db\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.243209 master-1 kubenswrapper[4771]: I1011 10:40:04.243115 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.243456 master-1 kubenswrapper[4771]: I1011 10:40:04.243424 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.243676 master-1 kubenswrapper[4771]: I1011 10:40:04.243650 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-b8cmk\" (UniqueName: \"kubernetes.io/projected/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-kube-api-access-b8cmk\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.243906 master-1 kubenswrapper[4771]: I1011 10:40:04.243879 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-main-tls\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.244087 master-1 kubenswrapper[4771]: I1011 10:40:04.244029 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-alertmanager-main-db\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.244221 master-1 kubenswrapper[4771]: I1011 10:40:04.244070 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-metrics-client-ca\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.244548 master-1 kubenswrapper[4771]: I1011 10:40:04.244505 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.244822 master-1 kubenswrapper[4771]: I1011 10:40:04.244784 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-config-volume\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.245069 master-1 kubenswrapper[4771]: I1011 10:40:04.245033 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-tls-assets\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.245315 master-1 kubenswrapper[4771]: I1011 10:40:04.245278 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-web-config\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.245667 master-1 kubenswrapper[4771]: I1011 10:40:04.244860 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-metrics-client-ca\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.245811 master-1 kubenswrapper[4771]: I1011 10:40:04.245668 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.245811 master-1 kubenswrapper[4771]: I1011 10:40:04.245593 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.250120 master-1 kubenswrapper[4771]: I1011 10:40:04.249984 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.250418 master-1 kubenswrapper[4771]: I1011 10:40:04.250315 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-main-tls\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.250418 master-1 kubenswrapper[4771]: I1011 10:40:04.250342 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-config-volume\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.250700 master-1 kubenswrapper[4771]: I1011 10:40:04.250650 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-config-out\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.250941 master-1 kubenswrapper[4771]: I1011 10:40:04.250880 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-tls-assets\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.251052 master-1 kubenswrapper[4771]: I1011 10:40:04.250980 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-web-config\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.252163 master-1 kubenswrapper[4771]: I1011 10:40:04.252119 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.254412 master-1 kubenswrapper[4771]: I1011 10:40:04.254306 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.263421 master-1 kubenswrapper[4771]: I1011 10:40:04.263338 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8cmk\" (UniqueName: \"kubernetes.io/projected/06b4c539-712c-4c8b-8b0f-ffbcbfd7811d-kube-api-access-b8cmk\") pod \"alertmanager-main-1\" (UID: \"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d\") " pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.336002 master-1 kubenswrapper[4771]: I1011 10:40:04.335946 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:04.817630 master-1 kubenswrapper[4771]: I1011 10:40:04.817534 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-1"] Oct 11 10:40:04.820143 master-1 kubenswrapper[4771]: W1011 10:40:04.820076 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b4c539_712c_4c8b_8b0f_ffbcbfd7811d.slice/crio-0e6f74ab535f56d067899045bb3191ead6cf63a23561a9074240cdae0614d5e1 WatchSource:0}: Error finding container 0e6f74ab535f56d067899045bb3191ead6cf63a23561a9074240cdae0614d5e1: Status 404 returned error can't find the container with id 0e6f74ab535f56d067899045bb3191ead6cf63a23561a9074240cdae0614d5e1 Oct 11 10:40:04.950506 master-1 kubenswrapper[4771]: I1011 10:40:04.950454 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f646dd4d8-v72dv"] Oct 11 10:40:04.952079 master-1 kubenswrapper[4771]: I1011 10:40:04.952054 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:04.955429 master-1 kubenswrapper[4771]: I1011 10:40:04.954903 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"thanos-querier-tls" Oct 11 10:40:04.955429 master-1 kubenswrapper[4771]: I1011 10:40:04.954904 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"thanos-querier-grpc-tls-92o819hatg7mp" Oct 11 10:40:04.955429 master-1 kubenswrapper[4771]: I1011 10:40:04.954957 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"thanos-querier-kube-rbac-proxy-rules" Oct 11 10:40:04.955429 master-1 kubenswrapper[4771]: I1011 10:40:04.955262 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"thanos-querier-kube-rbac-proxy-metrics" Oct 11 10:40:04.958271 master-1 kubenswrapper[4771]: I1011 10:40:04.956473 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"thanos-querier-kube-rbac-proxy" Oct 11 10:40:04.958271 master-1 kubenswrapper[4771]: I1011 10:40:04.956537 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"thanos-querier-kube-rbac-proxy-web" Oct 11 10:40:04.968926 master-1 kubenswrapper[4771]: I1011 10:40:04.968865 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f646dd4d8-v72dv"] Oct 11 10:40:05.056744 master-1 kubenswrapper[4771]: I1011 10:40:05.056675 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.056744 master-1 kubenswrapper[4771]: I1011 10:40:05.056734 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-grpc-tls\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.056744 master-1 kubenswrapper[4771]: I1011 10:40:05.056772 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.057565 master-1 kubenswrapper[4771]: I1011 10:40:05.056840 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-tls\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.057565 master-1 kubenswrapper[4771]: I1011 10:40:05.056900 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwfg\" (UniqueName: \"kubernetes.io/projected/2710b153-4085-41e5-8524-7cfb5d8c57f9-kube-api-access-rbwfg\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.057565 master-1 kubenswrapper[4771]: I1011 10:40:05.056941 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2710b153-4085-41e5-8524-7cfb5d8c57f9-metrics-client-ca\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.057565 master-1 kubenswrapper[4771]: I1011 10:40:05.056973 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.057565 master-1 kubenswrapper[4771]: I1011 10:40:05.057005 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.159098 master-1 kubenswrapper[4771]: I1011 10:40:05.158773 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-grpc-tls\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.159098 master-1 kubenswrapper[4771]: I1011 10:40:05.158873 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.159098 master-1 kubenswrapper[4771]: I1011 10:40:05.158933 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.159098 master-1 kubenswrapper[4771]: I1011 10:40:05.159015 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-tls\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.159098 master-1 kubenswrapper[4771]: I1011 10:40:05.159064 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwfg\" (UniqueName: \"kubernetes.io/projected/2710b153-4085-41e5-8524-7cfb5d8c57f9-kube-api-access-rbwfg\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.159798 master-1 kubenswrapper[4771]: I1011 10:40:05.159412 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2710b153-4085-41e5-8524-7cfb5d8c57f9-metrics-client-ca\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.160141 master-1 kubenswrapper[4771]: I1011 10:40:05.160076 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.160265 master-1 kubenswrapper[4771]: I1011 10:40:05.160149 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.161208 master-1 kubenswrapper[4771]: I1011 10:40:05.161157 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2710b153-4085-41e5-8524-7cfb5d8c57f9-metrics-client-ca\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.162808 master-1 kubenswrapper[4771]: I1011 10:40:05.162765 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.164002 master-1 kubenswrapper[4771]: I1011 10:40:05.163933 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.164160 master-1 kubenswrapper[4771]: I1011 10:40:05.164030 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-grpc-tls\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.164738 master-1 kubenswrapper[4771]: I1011 10:40:05.164692 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.165831 master-1 kubenswrapper[4771]: I1011 10:40:05.165753 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.165951 master-1 kubenswrapper[4771]: I1011 10:40:05.165890 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2710b153-4085-41e5-8524-7cfb5d8c57f9-secret-thanos-querier-tls\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.185286 master-1 kubenswrapper[4771]: I1011 10:40:05.185226 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwfg\" (UniqueName: \"kubernetes.io/projected/2710b153-4085-41e5-8524-7cfb5d8c57f9-kube-api-access-rbwfg\") pod \"thanos-querier-7f646dd4d8-v72dv\" (UID: \"2710b153-4085-41e5-8524-7cfb5d8c57f9\") " pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.270163 master-1 kubenswrapper[4771]: I1011 10:40:05.270089 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:05.369465 master-1 kubenswrapper[4771]: I1011 10:40:05.369344 4771 generic.go:334] "Generic (PLEG): container finished" podID="06b4c539-712c-4c8b-8b0f-ffbcbfd7811d" containerID="92ecee04176eac17c1c00567be2b537580d0d6115b2687d6cc4cbef9013df695" exitCode=0 Oct 11 10:40:05.369465 master-1 kubenswrapper[4771]: I1011 10:40:05.369455 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-1" event={"ID":"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d","Type":"ContainerDied","Data":"92ecee04176eac17c1c00567be2b537580d0d6115b2687d6cc4cbef9013df695"} Oct 11 10:40:05.369784 master-1 kubenswrapper[4771]: I1011 10:40:05.369496 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-1" event={"ID":"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d","Type":"ContainerStarted","Data":"0e6f74ab535f56d067899045bb3191ead6cf63a23561a9074240cdae0614d5e1"} Oct 11 10:40:05.739954 master-1 kubenswrapper[4771]: I1011 10:40:05.739885 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f646dd4d8-v72dv"] Oct 11 10:40:05.747272 master-1 kubenswrapper[4771]: W1011 10:40:05.747177 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2710b153_4085_41e5_8524_7cfb5d8c57f9.slice/crio-882cb261ffa89fd3372a5b4bceea1e3aa752e4db3b58b2a961b8efdf3a9b7438 WatchSource:0}: Error finding container 882cb261ffa89fd3372a5b4bceea1e3aa752e4db3b58b2a961b8efdf3a9b7438: Status 404 returned error can't find the container with id 882cb261ffa89fd3372a5b4bceea1e3aa752e4db3b58b2a961b8efdf3a9b7438 Oct 11 10:40:06.379409 master-1 kubenswrapper[4771]: I1011 10:40:06.379298 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" event={"ID":"2710b153-4085-41e5-8524-7cfb5d8c57f9","Type":"ContainerStarted","Data":"882cb261ffa89fd3372a5b4bceea1e3aa752e4db3b58b2a961b8efdf3a9b7438"} Oct 11 10:40:07.388540 master-1 kubenswrapper[4771]: I1011 10:40:07.388461 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-1" event={"ID":"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d","Type":"ContainerStarted","Data":"a76242e52f4101d9f26e218e8953c585b3982dfbdd75ea5689374d1e469d07d0"} Oct 11 10:40:07.388540 master-1 kubenswrapper[4771]: I1011 10:40:07.388524 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-1" event={"ID":"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d","Type":"ContainerStarted","Data":"6279c1fb3d44d787fe9e8e48fe2e0c0fc3d305e9cd737a3127a4af7da2544dc7"} Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: I1011 10:40:08.242827 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:08.243436 master-1 kubenswrapper[4771]: I1011 10:40:08.242912 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:08.398414 master-1 kubenswrapper[4771]: I1011 10:40:08.398300 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" event={"ID":"2710b153-4085-41e5-8524-7cfb5d8c57f9","Type":"ContainerStarted","Data":"acc2f488bf57141e6e0a36e89d0adfa78aa91304ff3f85a49628352123bd1320"} Oct 11 10:40:08.409012 master-1 kubenswrapper[4771]: I1011 10:40:08.408713 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-1" event={"ID":"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d","Type":"ContainerStarted","Data":"51ef77f2205ad388a0044218e2ad9456e242158c3929da3f14b908d1d952ac3d"} Oct 11 10:40:08.409012 master-1 kubenswrapper[4771]: I1011 10:40:08.408755 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-1" event={"ID":"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d","Type":"ContainerStarted","Data":"98498be567e03aba549687f3377ab610e9ad0ef0a1ff7cb07f31ae2f42c94fa6"} Oct 11 10:40:08.409012 master-1 kubenswrapper[4771]: I1011 10:40:08.408770 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-1" event={"ID":"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d","Type":"ContainerStarted","Data":"98daeb1d18320b2496b449e3dfeafab3b6fc9485441113346f32cdb7c9195430"} Oct 11 10:40:09.369009 master-1 kubenswrapper[4771]: I1011 10:40:09.368406 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-1"] Oct 11 10:40:09.370243 master-1 kubenswrapper[4771]: I1011 10:40:09.370186 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.375398 master-1 kubenswrapper[4771]: I1011 10:40:09.375344 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"kube-rbac-proxy" Oct 11 10:40:09.375752 master-1 kubenswrapper[4771]: I1011 10:40:09.375700 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"prometheus-k8s-tls" Oct 11 10:40:09.375876 master-1 kubenswrapper[4771]: I1011 10:40:09.375852 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"prometheus-k8s-grpc-tls-6sqva262urci3" Oct 11 10:40:09.376051 master-1 kubenswrapper[4771]: I1011 10:40:09.376003 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"prometheus-k8s" Oct 11 10:40:09.376173 master-1 kubenswrapper[4771]: I1011 10:40:09.376119 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"prometheus-k8s-web-config" Oct 11 10:40:09.376736 master-1 kubenswrapper[4771]: I1011 10:40:09.376681 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"prometheus-k8s-kube-rbac-proxy-web" Oct 11 10:40:09.377779 master-1 kubenswrapper[4771]: I1011 10:40:09.377704 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"serving-certs-ca-bundle" Oct 11 10:40:09.378556 master-1 kubenswrapper[4771]: I1011 10:40:09.378492 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"prometheus-k8s-tls-assets-0" Oct 11 10:40:09.378713 master-1 kubenswrapper[4771]: I1011 10:40:09.378618 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"prometheus-k8s-thanos-prometheus-http-client-file" Oct 11 10:40:09.378845 master-1 kubenswrapper[4771]: I1011 10:40:09.378797 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-monitoring"/"prometheus-k8s-thanos-sidecar-tls" Oct 11 10:40:09.393322 master-1 kubenswrapper[4771]: I1011 10:40:09.393213 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"prometheus-k8s-rulefiles-0" Oct 11 10:40:09.394397 master-1 kubenswrapper[4771]: I1011 10:40:09.394296 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-monitoring"/"prometheus-trusted-ca-bundle" Oct 11 10:40:09.398033 master-1 kubenswrapper[4771]: I1011 10:40:09.397965 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-1"] Oct 11 10:40:09.416993 master-1 kubenswrapper[4771]: I1011 10:40:09.416934 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" event={"ID":"2710b153-4085-41e5-8524-7cfb5d8c57f9","Type":"ContainerStarted","Data":"7ca2ecc0731a901c2fd6474f80cbed9283971035e0563cec27dc08282e199482"} Oct 11 10:40:09.416993 master-1 kubenswrapper[4771]: I1011 10:40:09.416982 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" event={"ID":"2710b153-4085-41e5-8524-7cfb5d8c57f9","Type":"ContainerStarted","Data":"7b3e36a7069b634ff3ca91c12f48e463cacec291be3dce74ba2bbd96106de635"} Oct 11 10:40:09.420500 master-1 kubenswrapper[4771]: I1011 10:40:09.420444 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.420584 master-1 kubenswrapper[4771]: I1011 10:40:09.420512 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bj9h\" (UniqueName: \"kubernetes.io/projected/e41e7753-f8a4-4f83-8061-b1610912b8e5-kube-api-access-4bj9h\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.420584 master-1 kubenswrapper[4771]: I1011 10:40:09.420548 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-web-config\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.420584 master-1 kubenswrapper[4771]: I1011 10:40:09.420578 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.420848 master-1 kubenswrapper[4771]: I1011 10:40:09.420609 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.420848 master-1 kubenswrapper[4771]: I1011 10:40:09.420665 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.420848 master-1 kubenswrapper[4771]: I1011 10:40:09.420698 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-configmap-metrics-client-ca\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.420848 master-1 kubenswrapper[4771]: I1011 10:40:09.420727 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e41e7753-f8a4-4f83-8061-b1610912b8e5-tls-assets\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.420848 master-1 kubenswrapper[4771]: I1011 10:40:09.420751 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.420848 master-1 kubenswrapper[4771]: I1011 10:40:09.420778 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.421542 master-1 kubenswrapper[4771]: I1011 10:40:09.421490 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e41e7753-f8a4-4f83-8061-b1610912b8e5-config-out\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.421754 master-1 kubenswrapper[4771]: I1011 10:40:09.421726 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.421997 master-1 kubenswrapper[4771]: I1011 10:40:09.421971 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e41e7753-f8a4-4f83-8061-b1610912b8e5-prometheus-k8s-db\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.422176 master-1 kubenswrapper[4771]: I1011 10:40:09.422150 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-config\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.422378 master-1 kubenswrapper[4771]: I1011 10:40:09.422329 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-metrics-client-certs\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.422596 master-1 kubenswrapper[4771]: I1011 10:40:09.422564 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.422776 master-1 kubenswrapper[4771]: I1011 10:40:09.422750 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.422957 master-1 kubenswrapper[4771]: I1011 10:40:09.422930 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-grpc-tls\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.524820 master-1 kubenswrapper[4771]: I1011 10:40:09.524750 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e41e7753-f8a4-4f83-8061-b1610912b8e5-prometheus-k8s-db\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525060 master-1 kubenswrapper[4771]: I1011 10:40:09.524836 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-config\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525060 master-1 kubenswrapper[4771]: I1011 10:40:09.524882 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-metrics-client-certs\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525060 master-1 kubenswrapper[4771]: I1011 10:40:09.524948 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525060 master-1 kubenswrapper[4771]: I1011 10:40:09.524965 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525060 master-1 kubenswrapper[4771]: I1011 10:40:09.524988 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-grpc-tls\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525060 master-1 kubenswrapper[4771]: I1011 10:40:09.525031 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525060 master-1 kubenswrapper[4771]: I1011 10:40:09.525049 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-4bj9h\" (UniqueName: \"kubernetes.io/projected/e41e7753-f8a4-4f83-8061-b1610912b8e5-kube-api-access-4bj9h\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525378 master-1 kubenswrapper[4771]: I1011 10:40:09.525077 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-web-config\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525378 master-1 kubenswrapper[4771]: I1011 10:40:09.525117 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525378 master-1 kubenswrapper[4771]: I1011 10:40:09.525135 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525378 master-1 kubenswrapper[4771]: I1011 10:40:09.525207 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525378 master-1 kubenswrapper[4771]: I1011 10:40:09.525242 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e41e7753-f8a4-4f83-8061-b1610912b8e5-prometheus-k8s-db\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525378 master-1 kubenswrapper[4771]: I1011 10:40:09.525275 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-configmap-metrics-client-ca\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525378 master-1 kubenswrapper[4771]: I1011 10:40:09.525344 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e41e7753-f8a4-4f83-8061-b1610912b8e5-tls-assets\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525775 master-1 kubenswrapper[4771]: I1011 10:40:09.525432 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525775 master-1 kubenswrapper[4771]: I1011 10:40:09.525467 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525775 master-1 kubenswrapper[4771]: I1011 10:40:09.525489 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e41e7753-f8a4-4f83-8061-b1610912b8e5-config-out\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.525775 master-1 kubenswrapper[4771]: I1011 10:40:09.525505 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.527158 master-1 kubenswrapper[4771]: I1011 10:40:09.527077 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.527907 master-1 kubenswrapper[4771]: I1011 10:40:09.527878 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-configmap-metrics-client-ca\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.528085 master-1 kubenswrapper[4771]: I1011 10:40:09.528038 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.528147 master-1 kubenswrapper[4771]: I1011 10:40:09.528087 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.529738 master-1 kubenswrapper[4771]: I1011 10:40:09.529671 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e41e7753-f8a4-4f83-8061-b1610912b8e5-config-out\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.530030 master-1 kubenswrapper[4771]: I1011 10:40:09.529981 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.530244 master-1 kubenswrapper[4771]: I1011 10:40:09.530188 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-grpc-tls\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.530658 master-1 kubenswrapper[4771]: I1011 10:40:09.530633 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.531303 master-1 kubenswrapper[4771]: I1011 10:40:09.531268 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-metrics-client-certs\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.531303 master-1 kubenswrapper[4771]: I1011 10:40:09.531275 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.531777 master-1 kubenswrapper[4771]: I1011 10:40:09.531735 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-config\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.531887 master-1 kubenswrapper[4771]: I1011 10:40:09.531834 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e41e7753-f8a4-4f83-8061-b1610912b8e5-tls-assets\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.532091 master-1 kubenswrapper[4771]: I1011 10:40:09.532054 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.532683 master-1 kubenswrapper[4771]: I1011 10:40:09.532640 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.533918 master-1 kubenswrapper[4771]: I1011 10:40:09.533877 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e41e7753-f8a4-4f83-8061-b1610912b8e5-web-config\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.537041 master-1 kubenswrapper[4771]: I1011 10:40:09.536998 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e41e7753-f8a4-4f83-8061-b1610912b8e5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.545434 master-1 kubenswrapper[4771]: I1011 10:40:09.545348 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bj9h\" (UniqueName: \"kubernetes.io/projected/e41e7753-f8a4-4f83-8061-b1610912b8e5-kube-api-access-4bj9h\") pod \"prometheus-k8s-1\" (UID: \"e41e7753-f8a4-4f83-8061-b1610912b8e5\") " pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:09.689340 master-1 kubenswrapper[4771]: I1011 10:40:09.688553 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:10.170754 master-1 kubenswrapper[4771]: I1011 10:40:10.170689 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-1"] Oct 11 10:40:10.171245 master-1 kubenswrapper[4771]: W1011 10:40:10.171198 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41e7753_f8a4_4f83_8061_b1610912b8e5.slice/crio-759b46a90e208c9d320ae5625ed991e26011bee2dd7849e5e8f3bffabdf513d2 WatchSource:0}: Error finding container 759b46a90e208c9d320ae5625ed991e26011bee2dd7849e5e8f3bffabdf513d2: Status 404 returned error can't find the container with id 759b46a90e208c9d320ae5625ed991e26011bee2dd7849e5e8f3bffabdf513d2 Oct 11 10:40:10.431164 master-1 kubenswrapper[4771]: I1011 10:40:10.430929 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-1" event={"ID":"06b4c539-712c-4c8b-8b0f-ffbcbfd7811d","Type":"ContainerStarted","Data":"2f30d6729b341dbdb9c05610aed3749be420a9b4f8bca413b81cd42c7a3fd58f"} Oct 11 10:40:10.434276 master-1 kubenswrapper[4771]: I1011 10:40:10.434215 4771 generic.go:334] "Generic (PLEG): container finished" podID="e41e7753-f8a4-4f83-8061-b1610912b8e5" containerID="8b9d35a5b980e617e2c3f966647e6a0a2866b6a35dbf6e74aaff90d59eccb2d5" exitCode=0 Oct 11 10:40:10.434443 master-1 kubenswrapper[4771]: I1011 10:40:10.434284 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-1" event={"ID":"e41e7753-f8a4-4f83-8061-b1610912b8e5","Type":"ContainerDied","Data":"8b9d35a5b980e617e2c3f966647e6a0a2866b6a35dbf6e74aaff90d59eccb2d5"} Oct 11 10:40:10.434443 master-1 kubenswrapper[4771]: I1011 10:40:10.434399 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-1" event={"ID":"e41e7753-f8a4-4f83-8061-b1610912b8e5","Type":"ContainerStarted","Data":"759b46a90e208c9d320ae5625ed991e26011bee2dd7849e5e8f3bffabdf513d2"} Oct 11 10:40:10.449662 master-1 kubenswrapper[4771]: I1011 10:40:10.449579 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" event={"ID":"2710b153-4085-41e5-8524-7cfb5d8c57f9","Type":"ContainerStarted","Data":"b6c95c59cc9b33b208e816806002297456cf1628f1011955a6a7777e48e09342"} Oct 11 10:40:10.449662 master-1 kubenswrapper[4771]: I1011 10:40:10.449662 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:10.449885 master-1 kubenswrapper[4771]: I1011 10:40:10.449684 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" event={"ID":"2710b153-4085-41e5-8524-7cfb5d8c57f9","Type":"ContainerStarted","Data":"73b24c57422a3f7f43d64b8eb582bbcc3d940da1dcdc6d9d55cdff5cbcf0c8d4"} Oct 11 10:40:10.449885 master-1 kubenswrapper[4771]: I1011 10:40:10.449705 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" event={"ID":"2710b153-4085-41e5-8524-7cfb5d8c57f9","Type":"ContainerStarted","Data":"963b466d2923c869692640d42b7341628728b839587f191ba41778719ee555ef"} Oct 11 10:40:10.467838 master-1 kubenswrapper[4771]: I1011 10:40:10.467512 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-1" podStartSLOduration=3.423964435 podStartE2EDuration="7.467501409s" podCreationTimestamp="2025-10-11 10:40:03 +0000 UTC" firstStartedPulling="2025-10-11 10:40:05.37133028 +0000 UTC m=+837.345556751" lastFinishedPulling="2025-10-11 10:40:09.414867274 +0000 UTC m=+841.389093725" observedRunningTime="2025-10-11 10:40:10.462666489 +0000 UTC m=+842.436892990" watchObservedRunningTime="2025-10-11 10:40:10.467501409 +0000 UTC m=+842.441727850" Oct 11 10:40:10.502410 master-1 kubenswrapper[4771]: I1011 10:40:10.502254 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" podStartSLOduration=2.834592451 podStartE2EDuration="6.50222628s" podCreationTimestamp="2025-10-11 10:40:04 +0000 UTC" firstStartedPulling="2025-10-11 10:40:05.751624962 +0000 UTC m=+837.725851443" lastFinishedPulling="2025-10-11 10:40:09.419258821 +0000 UTC m=+841.393485272" observedRunningTime="2025-10-11 10:40:10.489988384 +0000 UTC m=+842.464214865" watchObservedRunningTime="2025-10-11 10:40:10.50222628 +0000 UTC m=+842.476452761" Oct 11 10:40:10.995300 master-1 kubenswrapper[4771]: I1011 10:40:10.995191 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b"] Oct 11 10:40:10.995709 master-1 kubenswrapper[4771]: I1011 10:40:10.995618 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" containerID="cri-o://db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217" gracePeriod=120 Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: I1011 10:40:11.144408 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:11.144475 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:11.145481 master-1 kubenswrapper[4771]: I1011 10:40:11.144537 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:12.580558 master-1 kubenswrapper[4771]: I1011 10:40:12.580448 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:40:12.581284 master-1 kubenswrapper[4771]: I1011 10:40:12.580608 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: I1011 10:40:13.242182 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:13.242263 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:13.243893 master-1 kubenswrapper[4771]: I1011 10:40:13.242277 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:14.339380 master-1 kubenswrapper[4771]: I1011 10:40:14.336429 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:14.467105 master-1 kubenswrapper[4771]: I1011 10:40:14.467025 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-1" event={"ID":"e41e7753-f8a4-4f83-8061-b1610912b8e5","Type":"ContainerStarted","Data":"94398d93e05baa7ebaf4c98738bf8458b8c61d13c1518d588a7b6e6ba51ca470"} Oct 11 10:40:14.467105 master-1 kubenswrapper[4771]: I1011 10:40:14.467088 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-1" event={"ID":"e41e7753-f8a4-4f83-8061-b1610912b8e5","Type":"ContainerStarted","Data":"cfb4a3a7730430abd44292ac93b9633e1d847284a2ef917109d0511f065fd804"} Oct 11 10:40:14.467380 master-1 kubenswrapper[4771]: I1011 10:40:14.467121 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-1" event={"ID":"e41e7753-f8a4-4f83-8061-b1610912b8e5","Type":"ContainerStarted","Data":"759fc712555322c0af22165b0ad340ddaafbf49dac9c9493241bfd8f0d65744f"} Oct 11 10:40:15.282014 master-1 kubenswrapper[4771]: I1011 10:40:15.281938 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f646dd4d8-v72dv" Oct 11 10:40:15.480921 master-1 kubenswrapper[4771]: I1011 10:40:15.480844 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-1" event={"ID":"e41e7753-f8a4-4f83-8061-b1610912b8e5","Type":"ContainerStarted","Data":"5cb08c81b77eb576dca7c07741e584389136fa4bff5ab56041056a7455f7b9f7"} Oct 11 10:40:15.480921 master-1 kubenswrapper[4771]: I1011 10:40:15.480897 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-1" event={"ID":"e41e7753-f8a4-4f83-8061-b1610912b8e5","Type":"ContainerStarted","Data":"17632de8110e3b81a16e263c87b729e2324b6ff74c3372023897bfeab09634b0"} Oct 11 10:40:15.480921 master-1 kubenswrapper[4771]: I1011 10:40:15.480913 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-1" event={"ID":"e41e7753-f8a4-4f83-8061-b1610912b8e5","Type":"ContainerStarted","Data":"d299a1f483193709c3f90f4daf69e455c6ac2fa5e39b9379cbcae39ed442cb31"} Oct 11 10:40:15.529409 master-1 kubenswrapper[4771]: I1011 10:40:15.529276 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-1" podStartSLOduration=3.105603119 podStartE2EDuration="6.529253534s" podCreationTimestamp="2025-10-11 10:40:09 +0000 UTC" firstStartedPulling="2025-10-11 10:40:10.436540469 +0000 UTC m=+842.410766940" lastFinishedPulling="2025-10-11 10:40:13.860190914 +0000 UTC m=+845.834417355" observedRunningTime="2025-10-11 10:40:15.522155747 +0000 UTC m=+847.496382238" watchObservedRunningTime="2025-10-11 10:40:15.529253534 +0000 UTC m=+847.503479995" Oct 11 10:40:16.116593 master-1 kubenswrapper[4771]: I1011 10:40:16.116468 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-apiserver/apiserver-7845cf54d8-g8x5z"] Oct 11 10:40:16.117034 master-1 kubenswrapper[4771]: I1011 10:40:16.116912 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" containerID="cri-o://2ccd5ea4ca8c2b32e04ef7419d2c1c1ac0971dd1b18e1a37cd16058b70e5a98c" gracePeriod=120 Oct 11 10:40:16.117258 master-1 kubenswrapper[4771]: I1011 10:40:16.117072 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver-check-endpoints" containerID="cri-o://a0772db7a40ce6f228f65f235a6668a5f2f1781a4f227000cf9ad01206d856f2" gracePeriod=120 Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: I1011 10:40:16.143830 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:16.143898 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:16.144541 master-1 kubenswrapper[4771]: I1011 10:40:16.143923 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:16.493351 master-1 kubenswrapper[4771]: I1011 10:40:16.493084 4771 generic.go:334] "Generic (PLEG): container finished" podID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerID="a0772db7a40ce6f228f65f235a6668a5f2f1781a4f227000cf9ad01206d856f2" exitCode=0 Oct 11 10:40:16.493351 master-1 kubenswrapper[4771]: I1011 10:40:16.493176 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" event={"ID":"a2bf529d-094c-4406-8ce6-890cf8c0b840","Type":"ContainerDied","Data":"a0772db7a40ce6f228f65f235a6668a5f2f1781a4f227000cf9ad01206d856f2"} Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: I1011 10:40:17.258610 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:17.258700 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:17.260613 master-1 kubenswrapper[4771]: I1011 10:40:17.258713 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: I1011 10:40:18.245029 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:18.245108 master-1 kubenswrapper[4771]: I1011 10:40:18.245093 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:19.689040 master-1 kubenswrapper[4771]: I1011 10:40:19.688951 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: I1011 10:40:21.140867 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:21.140927 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:21.141939 master-1 kubenswrapper[4771]: I1011 10:40:21.141608 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:21.141939 master-1 kubenswrapper[4771]: I1011 10:40:21.141810 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: I1011 10:40:22.255566 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:22.255612 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:22.256933 master-1 kubenswrapper[4771]: I1011 10:40:22.256513 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:22.581168 master-1 kubenswrapper[4771]: I1011 10:40:22.581075 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:40:22.581168 master-1 kubenswrapper[4771]: I1011 10:40:22.581156 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: I1011 10:40:23.245071 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:23.245171 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:23.247142 master-1 kubenswrapper[4771]: I1011 10:40:23.245175 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: I1011 10:40:26.143279 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:26.143403 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:26.144632 master-1 kubenswrapper[4771]: I1011 10:40:26.143586 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: I1011 10:40:27.257386 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:27.257479 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:27.258705 master-1 kubenswrapper[4771]: I1011 10:40:27.257507 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:27.258705 master-1 kubenswrapper[4771]: I1011 10:40:27.257648 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: I1011 10:40:28.244205 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:28.244326 master-1 kubenswrapper[4771]: I1011 10:40:28.244296 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: I1011 10:40:31.143277 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:31.143377 master-1 kubenswrapper[4771]: I1011 10:40:31.143372 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: I1011 10:40:32.254524 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:32.254590 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:32.255649 master-1 kubenswrapper[4771]: I1011 10:40:32.254601 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:32.581956 master-1 kubenswrapper[4771]: I1011 10:40:32.581842 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:40:32.582307 master-1 kubenswrapper[4771]: I1011 10:40:32.581987 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: I1011 10:40:33.245314 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:33.245455 master-1 kubenswrapper[4771]: I1011 10:40:33.245425 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:34.382880 master-1 kubenswrapper[4771]: I1011 10:40:34.382771 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/alertmanager-main-1" Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: I1011 10:40:36.145949 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:36.146034 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:36.147573 master-1 kubenswrapper[4771]: I1011 10:40:36.146047 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: I1011 10:40:37.257723 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:37.257890 master-1 kubenswrapper[4771]: I1011 10:40:37.257833 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: I1011 10:40:38.243650 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:38.243750 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:38.246634 master-1 kubenswrapper[4771]: I1011 10:40:38.243762 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: I1011 10:40:41.142519 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:41.142701 master-1 kubenswrapper[4771]: I1011 10:40:41.142615 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: I1011 10:40:42.256430 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:42.256521 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:42.257447 master-1 kubenswrapper[4771]: I1011 10:40:42.256521 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:42.580540 master-1 kubenswrapper[4771]: I1011 10:40:42.580469 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:40:42.580780 master-1 kubenswrapper[4771]: I1011 10:40:42.580556 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: I1011 10:40:43.244915 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:43.245000 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:43.247998 master-1 kubenswrapper[4771]: I1011 10:40:43.245075 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: I1011 10:40:46.144614 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:46.144739 master-1 kubenswrapper[4771]: I1011 10:40:46.144728 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: I1011 10:40:47.257496 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:47.257650 master-1 kubenswrapper[4771]: I1011 10:40:47.257592 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: I1011 10:40:48.247664 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:48.247756 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:48.251617 master-1 kubenswrapper[4771]: I1011 10:40:48.247763 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: I1011 10:40:51.144491 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:51.144578 master-1 kubenswrapper[4771]: I1011 10:40:51.144568 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: I1011 10:40:52.255091 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:52.255191 master-1 kubenswrapper[4771]: I1011 10:40:52.255171 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:52.581520 master-1 kubenswrapper[4771]: I1011 10:40:52.581441 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:40:52.581854 master-1 kubenswrapper[4771]: I1011 10:40:52.581541 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:40:53.239211 master-1 kubenswrapper[4771]: I1011 10:40:53.238558 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:40:53.239211 master-1 kubenswrapper[4771]: I1011 10:40:53.238662 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:40:54.966649 master-1 kubenswrapper[4771]: I1011 10:40:54.966584 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_e39186c2ebd02622803bdbec6984de2a/kube-apiserver-cert-syncer/0.log" Oct 11 10:40:54.967778 master-1 kubenswrapper[4771]: I1011 10:40:54.967667 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:40:54.972974 master-1 kubenswrapper[4771]: I1011 10:40:54.972917 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="e39186c2ebd02622803bdbec6984de2a" podUID="42d61efaa0f96869cf2939026aad6022" Oct 11 10:40:55.030316 master-1 kubenswrapper[4771]: I1011 10:40:55.030181 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-cert-dir\") pod \"e39186c2ebd02622803bdbec6984de2a\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " Oct 11 10:40:55.030316 master-1 kubenswrapper[4771]: I1011 10:40:55.030306 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-audit-dir\") pod \"e39186c2ebd02622803bdbec6984de2a\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " Oct 11 10:40:55.030845 master-1 kubenswrapper[4771]: I1011 10:40:55.030397 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-cert-dir" (OuterVolumeSpecName: "cert-dir") pod "e39186c2ebd02622803bdbec6984de2a" (UID: "e39186c2ebd02622803bdbec6984de2a"). InnerVolumeSpecName "cert-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:40:55.030845 master-1 kubenswrapper[4771]: I1011 10:40:55.030450 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-resource-dir\") pod \"e39186c2ebd02622803bdbec6984de2a\" (UID: \"e39186c2ebd02622803bdbec6984de2a\") " Oct 11 10:40:55.030845 master-1 kubenswrapper[4771]: I1011 10:40:55.030501 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "e39186c2ebd02622803bdbec6984de2a" (UID: "e39186c2ebd02622803bdbec6984de2a"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:40:55.030845 master-1 kubenswrapper[4771]: I1011 10:40:55.030551 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-resource-dir" (OuterVolumeSpecName: "resource-dir") pod "e39186c2ebd02622803bdbec6984de2a" (UID: "e39186c2ebd02622803bdbec6984de2a"). InnerVolumeSpecName "resource-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:40:55.031320 master-1 kubenswrapper[4771]: I1011 10:40:55.030901 4771 reconciler_common.go:293] "Volume detached for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-resource-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:40:55.031320 master-1 kubenswrapper[4771]: I1011 10:40:55.030935 4771 reconciler_common.go:293] "Volume detached for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-cert-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:40:55.031320 master-1 kubenswrapper[4771]: I1011 10:40:55.030961 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/e39186c2ebd02622803bdbec6984de2a-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:40:55.792628 master-1 kubenswrapper[4771]: I1011 10:40:55.792531 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_e39186c2ebd02622803bdbec6984de2a/kube-apiserver-cert-syncer/0.log" Oct 11 10:40:55.793910 master-1 kubenswrapper[4771]: I1011 10:40:55.793839 4771 generic.go:334] "Generic (PLEG): container finished" podID="e39186c2ebd02622803bdbec6984de2a" containerID="be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a" exitCode=0 Oct 11 10:40:55.794022 master-1 kubenswrapper[4771]: I1011 10:40:55.793960 4771 scope.go:117] "RemoveContainer" containerID="50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07" Oct 11 10:40:55.794022 master-1 kubenswrapper[4771]: I1011 10:40:55.793992 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:40:55.800791 master-1 kubenswrapper[4771]: I1011 10:40:55.800717 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="e39186c2ebd02622803bdbec6984de2a" podUID="42d61efaa0f96869cf2939026aad6022" Oct 11 10:40:55.815969 master-1 kubenswrapper[4771]: I1011 10:40:55.814351 4771 scope.go:117] "RemoveContainer" containerID="637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4" Oct 11 10:40:55.831950 master-1 kubenswrapper[4771]: I1011 10:40:55.831788 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="e39186c2ebd02622803bdbec6984de2a" podUID="42d61efaa0f96869cf2939026aad6022" Oct 11 10:40:55.838945 master-1 kubenswrapper[4771]: I1011 10:40:55.838893 4771 scope.go:117] "RemoveContainer" containerID="7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109" Oct 11 10:40:55.858582 master-1 kubenswrapper[4771]: I1011 10:40:55.858525 4771 scope.go:117] "RemoveContainer" containerID="0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd" Oct 11 10:40:55.878610 master-1 kubenswrapper[4771]: I1011 10:40:55.878566 4771 scope.go:117] "RemoveContainer" containerID="be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a" Oct 11 10:40:55.901842 master-1 kubenswrapper[4771]: I1011 10:40:55.901785 4771 scope.go:117] "RemoveContainer" containerID="70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912" Oct 11 10:40:55.927112 master-1 kubenswrapper[4771]: I1011 10:40:55.926950 4771 scope.go:117] "RemoveContainer" containerID="50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07" Oct 11 10:40:55.927771 master-1 kubenswrapper[4771]: E1011 10:40:55.927701 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07\": container with ID starting with 50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07 not found: ID does not exist" containerID="50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07" Oct 11 10:40:55.927835 master-1 kubenswrapper[4771]: I1011 10:40:55.927785 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07"} err="failed to get container status \"50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07\": rpc error: code = NotFound desc = could not find container \"50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07\": container with ID starting with 50a31e36a673a3d932b0eed3747b62b95f6a9f4bd14409954bbb8b619a64ca07 not found: ID does not exist" Oct 11 10:40:55.927896 master-1 kubenswrapper[4771]: I1011 10:40:55.927836 4771 scope.go:117] "RemoveContainer" containerID="637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4" Oct 11 10:40:55.928524 master-1 kubenswrapper[4771]: E1011 10:40:55.928444 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4\": container with ID starting with 637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4 not found: ID does not exist" containerID="637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4" Oct 11 10:40:55.928599 master-1 kubenswrapper[4771]: I1011 10:40:55.928542 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4"} err="failed to get container status \"637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4\": rpc error: code = NotFound desc = could not find container \"637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4\": container with ID starting with 637d81cee36b54a2c10877824c0d4c8cd57b4ef94675651e76c7ca2c91addea4 not found: ID does not exist" Oct 11 10:40:55.928653 master-1 kubenswrapper[4771]: I1011 10:40:55.928607 4771 scope.go:117] "RemoveContainer" containerID="7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109" Oct 11 10:40:55.929157 master-1 kubenswrapper[4771]: E1011 10:40:55.929101 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109\": container with ID starting with 7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109 not found: ID does not exist" containerID="7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109" Oct 11 10:40:55.929219 master-1 kubenswrapper[4771]: I1011 10:40:55.929155 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109"} err="failed to get container status \"7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109\": rpc error: code = NotFound desc = could not find container \"7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109\": container with ID starting with 7808e9129366cc3f545a8ddafc086a76064c7552891a59b63333320814e12109 not found: ID does not exist" Oct 11 10:40:55.929219 master-1 kubenswrapper[4771]: I1011 10:40:55.929187 4771 scope.go:117] "RemoveContainer" containerID="0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd" Oct 11 10:40:55.929834 master-1 kubenswrapper[4771]: E1011 10:40:55.929773 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd\": container with ID starting with 0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd not found: ID does not exist" containerID="0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd" Oct 11 10:40:55.929903 master-1 kubenswrapper[4771]: I1011 10:40:55.929836 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd"} err="failed to get container status \"0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd\": rpc error: code = NotFound desc = could not find container \"0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd\": container with ID starting with 0e245a1193b14427c4e53af18fc7d7caad0bae63a50c8f3671564cc8eb9cc3dd not found: ID does not exist" Oct 11 10:40:55.929903 master-1 kubenswrapper[4771]: I1011 10:40:55.929878 4771 scope.go:117] "RemoveContainer" containerID="be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a" Oct 11 10:40:55.930503 master-1 kubenswrapper[4771]: E1011 10:40:55.930457 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a\": container with ID starting with be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a not found: ID does not exist" containerID="be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a" Oct 11 10:40:55.930566 master-1 kubenswrapper[4771]: I1011 10:40:55.930504 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a"} err="failed to get container status \"be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a\": rpc error: code = NotFound desc = could not find container \"be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a\": container with ID starting with be8c3f5de1224f1ddb95dff091a6c317c3ddd56bd6bebec9f107f4b1c1bd098a not found: ID does not exist" Oct 11 10:40:55.930566 master-1 kubenswrapper[4771]: I1011 10:40:55.930539 4771 scope.go:117] "RemoveContainer" containerID="70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912" Oct 11 10:40:55.931050 master-1 kubenswrapper[4771]: E1011 10:40:55.930970 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912\": container with ID starting with 70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912 not found: ID does not exist" containerID="70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912" Oct 11 10:40:55.931114 master-1 kubenswrapper[4771]: I1011 10:40:55.931051 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912"} err="failed to get container status \"70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912\": rpc error: code = NotFound desc = could not find container \"70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912\": container with ID starting with 70b3ac8371a68d1ef8731071faccdda868469d141980d26eb4114af177d58912 not found: ID does not exist" Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: I1011 10:40:56.144610 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:56.144709 master-1 kubenswrapper[4771]: I1011 10:40:56.144702 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:56.450295 master-1 kubenswrapper[4771]: I1011 10:40:56.450151 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e39186c2ebd02622803bdbec6984de2a" path="/var/lib/kubelet/pods/e39186c2ebd02622803bdbec6984de2a/volumes" Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: I1011 10:40:57.257431 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:40:57.257535 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:40:57.259972 master-1 kubenswrapper[4771]: I1011 10:40:57.257538 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:40:58.238010 master-1 kubenswrapper[4771]: I1011 10:40:58.237936 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:40:58.238396 master-1 kubenswrapper[4771]: I1011 10:40:58.238010 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:41:00.437651 master-1 kubenswrapper[4771]: I1011 10:41:00.437553 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:41:00.459204 master-1 kubenswrapper[4771]: I1011 10:41:00.459170 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="233d76fa-d8e2-41eb-9272-6cdd0056b793" Oct 11 10:41:00.459328 master-1 kubenswrapper[4771]: I1011 10:41:00.459316 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="233d76fa-d8e2-41eb-9272-6cdd0056b793" Oct 11 10:41:00.477904 master-1 kubenswrapper[4771]: I1011 10:41:00.477860 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:41:00.516601 master-1 kubenswrapper[4771]: I1011 10:41:00.516536 4771 kubelet.go:1914] "Deleted mirror pod because it is outdated" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:41:00.520186 master-1 kubenswrapper[4771]: I1011 10:41:00.520141 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:41:00.539027 master-1 kubenswrapper[4771]: I1011 10:41:00.538952 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:41:00.544485 master-1 kubenswrapper[4771]: I1011 10:41:00.544414 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:41:00.568078 master-1 kubenswrapper[4771]: W1011 10:41:00.568005 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d61efaa0f96869cf2939026aad6022.slice/crio-61d36eba775f4bff620711b60052065dee1d2c15d7f0fa86d827b5f7e6631d7d WatchSource:0}: Error finding container 61d36eba775f4bff620711b60052065dee1d2c15d7f0fa86d827b5f7e6631d7d: Status 404 returned error can't find the container with id 61d36eba775f4bff620711b60052065dee1d2c15d7f0fa86d827b5f7e6631d7d Oct 11 10:41:00.834303 master-1 kubenswrapper[4771]: I1011 10:41:00.834153 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"42d61efaa0f96869cf2939026aad6022","Type":"ContainerStarted","Data":"61d36eba775f4bff620711b60052065dee1d2c15d7f0fa86d827b5f7e6631d7d"} Oct 11 10:41:01.138320 master-1 kubenswrapper[4771]: I1011 10:41:01.138183 4771 patch_prober.go:28] interesting pod/apiserver-68f4c55ff4-z898b container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.51:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.51:8443: connect: connection refused" start-of-body= Oct 11 10:41:01.138320 master-1 kubenswrapper[4771]: I1011 10:41:01.138278 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" probeResult="failure" output="Get \"https://10.129.0.51:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.51:8443: connect: connection refused" Oct 11 10:41:01.845158 master-1 kubenswrapper[4771]: I1011 10:41:01.843987 4771 generic.go:334] "Generic (PLEG): container finished" podID="42d61efaa0f96869cf2939026aad6022" containerID="546001aeab4a76f01af18f5f0a0232cc48a20c2025802d7d9983eb8c840e0866" exitCode=0 Oct 11 10:41:01.845158 master-1 kubenswrapper[4771]: I1011 10:41:01.844062 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"42d61efaa0f96869cf2939026aad6022","Type":"ContainerDied","Data":"546001aeab4a76f01af18f5f0a0232cc48a20c2025802d7d9983eb8c840e0866"} Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: I1011 10:41:02.262602 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/image.openshift.io-apiserver-caches ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-bootstrapclusterroles ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectcache ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/project.openshift.io-projectauthorizationcache ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startinformers ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-restmapperupdater ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:41:02.262903 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:41:02.263654 master-1 kubenswrapper[4771]: I1011 10:41:02.263087 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:41:02.580293 master-1 kubenswrapper[4771]: I1011 10:41:02.580205 4771 patch_prober.go:28] interesting pod/console-775ff6c4fc-csp4z container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" start-of-body= Oct 11 10:41:02.580293 master-1 kubenswrapper[4771]: I1011 10:41:02.580289 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" probeResult="failure" output="Get \"https://10.129.0.73:8443/health\": dial tcp 10.129.0.73:8443: connect: connection refused" Oct 11 10:41:02.854164 master-1 kubenswrapper[4771]: I1011 10:41:02.854095 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"42d61efaa0f96869cf2939026aad6022","Type":"ContainerStarted","Data":"55ecf6fefa862d92619ce534057ad20c836371d13f4c0d70468214b0bd6e3db4"} Oct 11 10:41:02.854164 master-1 kubenswrapper[4771]: I1011 10:41:02.854157 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"42d61efaa0f96869cf2939026aad6022","Type":"ContainerStarted","Data":"7e5a3711f36461fe4ced62a6738267cdf151c6f22d750936a4256bced2e89c2a"} Oct 11 10:41:02.854164 master-1 kubenswrapper[4771]: I1011 10:41:02.854175 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"42d61efaa0f96869cf2939026aad6022","Type":"ContainerStarted","Data":"d035b13d9431b1216e273c4ac7fb5eb87624d8740b70d29326082336302e3b46"} Oct 11 10:41:03.427210 master-1 kubenswrapper[4771]: I1011 10:41:03.427147 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.468227 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-client\") pod \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.468313 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-trusted-ca-bundle\") pod \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.468342 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-serving-ca\") pod \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.468381 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-policies\") pod \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.468437 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-encryption-config\") pod \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.468465 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crdvt\" (UniqueName: \"kubernetes.io/projected/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-kube-api-access-crdvt\") pod \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.468505 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-dir\") pod \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.468533 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-serving-cert\") pod \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\" (UID: \"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40\") " Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.470177 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-policies" (OuterVolumeSpecName: "audit-policies") pod "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" (UID: "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40"). InnerVolumeSpecName "audit-policies". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.474100 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-client" (OuterVolumeSpecName: "etcd-client") pod "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" (UID: "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40"). InnerVolumeSpecName "etcd-client". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.474469 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" (UID: "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:41:03.475449 master-1 kubenswrapper[4771]: I1011 10:41:03.474796 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-serving-ca" (OuterVolumeSpecName: "etcd-serving-ca") pod "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" (UID: "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40"). InnerVolumeSpecName "etcd-serving-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:41:03.478244 master-1 kubenswrapper[4771]: I1011 10:41:03.478189 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-kube-api-access-crdvt" (OuterVolumeSpecName: "kube-api-access-crdvt") pod "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" (UID: "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40"). InnerVolumeSpecName "kube-api-access-crdvt". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:41:03.478320 master-1 kubenswrapper[4771]: I1011 10:41:03.478269 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" (UID: "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:41:03.480633 master-1 kubenswrapper[4771]: I1011 10:41:03.480593 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-encryption-config" (OuterVolumeSpecName: "encryption-config") pod "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" (UID: "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40"). InnerVolumeSpecName "encryption-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:41:03.482274 master-1 kubenswrapper[4771]: I1011 10:41:03.482204 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" (UID: "44bb7164-0bee-4f90-8bf6-a2d73e1f3d40"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:41:03.570820 master-1 kubenswrapper[4771]: I1011 10:41:03.570539 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:41:03.570820 master-1 kubenswrapper[4771]: I1011 10:41:03.570755 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-serving-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:03.570820 master-1 kubenswrapper[4771]: I1011 10:41:03.570772 4771 reconciler_common.go:293] "Volume detached for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-policies\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:03.570820 master-1 kubenswrapper[4771]: I1011 10:41:03.570784 4771 reconciler_common.go:293] "Volume detached for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-encryption-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:03.570820 master-1 kubenswrapper[4771]: I1011 10:41:03.570795 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-crdvt\" (UniqueName: \"kubernetes.io/projected/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-kube-api-access-crdvt\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:03.570820 master-1 kubenswrapper[4771]: I1011 10:41:03.570809 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:03.570820 master-1 kubenswrapper[4771]: I1011 10:41:03.570820 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:03.570820 master-1 kubenswrapper[4771]: I1011 10:41:03.570829 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-etcd-client\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:03.570820 master-1 kubenswrapper[4771]: I1011 10:41:03.570838 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:03.571454 master-1 kubenswrapper[4771]: I1011 10:41:03.570936 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/d7647696-42d9-4dd9-bc3b-a4d52a42cf9a-etc-docker\") pod \"operator-controller-controller-manager-668cb7cdc8-bqdlc\" (UID: \"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a\") " pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:41:03.664741 master-1 kubenswrapper[4771]: I1011 10:41:03.664635 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:41:03.672641 master-1 kubenswrapper[4771]: I1011 10:41:03.672551 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:41:03.672932 master-1 kubenswrapper[4771]: I1011 10:41:03.672896 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-docker\" (UniqueName: \"kubernetes.io/host-path/6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b-etc-docker\") pod \"catalogd-controller-manager-596f9d8bbf-tpzsm\" (UID: \"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b\") " pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:41:03.737547 master-1 kubenswrapper[4771]: I1011 10:41:03.737261 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 403" start-of-body={"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/readyz\": RBAC: [clusterrole.rbac.authorization.k8s.io \"system:openshift:public-info-viewer\" not found, clusterrole.rbac.authorization.k8s.io \"system:public-info-viewer\" not found]","reason":"Forbidden","details":{},"code":403} Oct 11 10:41:03.738227 master-1 kubenswrapper[4771]: I1011 10:41:03.737902 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 403" Oct 11 10:41:03.879619 master-1 kubenswrapper[4771]: I1011 10:41:03.879563 4771 generic.go:334] "Generic (PLEG): container finished" podID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerID="db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217" exitCode=0 Oct 11 10:41:03.880076 master-1 kubenswrapper[4771]: I1011 10:41:03.879666 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" event={"ID":"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40","Type":"ContainerDied","Data":"db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217"} Oct 11 10:41:03.880076 master-1 kubenswrapper[4771]: I1011 10:41:03.879695 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" event={"ID":"44bb7164-0bee-4f90-8bf6-a2d73e1f3d40","Type":"ContainerDied","Data":"ef282371271fc7902dfe16d939904e98053b587f042204eef235e27cd9b5b8b6"} Oct 11 10:41:03.880076 master-1 kubenswrapper[4771]: I1011 10:41:03.879716 4771 scope.go:117] "RemoveContainer" containerID="db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217" Oct 11 10:41:03.880076 master-1 kubenswrapper[4771]: I1011 10:41:03.879839 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b" Oct 11 10:41:03.886210 master-1 kubenswrapper[4771]: I1011 10:41:03.886134 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"42d61efaa0f96869cf2939026aad6022","Type":"ContainerStarted","Data":"a15e7539d2a0c42e8c6c8995bf98ff26ca0f322daf83394df48b4f13fc42d10b"} Oct 11 10:41:03.886210 master-1 kubenswrapper[4771]: I1011 10:41:03.886211 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"42d61efaa0f96869cf2939026aad6022","Type":"ContainerStarted","Data":"452189c1a156cff2357db3338f99f86d41c76ed0f97b4459672ad6a8fe0dc5c7"} Oct 11 10:41:03.886826 master-1 kubenswrapper[4771]: I1011 10:41:03.886797 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:41:03.921880 master-1 kubenswrapper[4771]: I1011 10:41:03.913798 4771 scope.go:117] "RemoveContainer" containerID="099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61" Oct 11 10:41:03.946640 master-1 kubenswrapper[4771]: I1011 10:41:03.946563 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/kube-apiserver-master-1" podStartSLOduration=3.946543781 podStartE2EDuration="3.946543781s" podCreationTimestamp="2025-10-11 10:41:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:41:03.927159198 +0000 UTC m=+895.901385659" watchObservedRunningTime="2025-10-11 10:41:03.946543781 +0000 UTC m=+895.920770222" Oct 11 10:41:03.949172 master-1 kubenswrapper[4771]: I1011 10:41:03.949125 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b"] Oct 11 10:41:03.952151 master-1 kubenswrapper[4771]: I1011 10:41:03.951926 4771 scope.go:117] "RemoveContainer" containerID="db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217" Oct 11 10:41:03.952620 master-1 kubenswrapper[4771]: E1011 10:41:03.952588 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217\": container with ID starting with db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217 not found: ID does not exist" containerID="db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217" Oct 11 10:41:03.952691 master-1 kubenswrapper[4771]: I1011 10:41:03.952629 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217"} err="failed to get container status \"db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217\": rpc error: code = NotFound desc = could not find container \"db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217\": container with ID starting with db0964db198d321448b29e5ac3377a039eb46842494e887c86878f66ad14d217 not found: ID does not exist" Oct 11 10:41:03.952691 master-1 kubenswrapper[4771]: I1011 10:41:03.952658 4771 scope.go:117] "RemoveContainer" containerID="099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61" Oct 11 10:41:03.953408 master-1 kubenswrapper[4771]: E1011 10:41:03.953335 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61\": container with ID starting with 099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61 not found: ID does not exist" containerID="099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61" Oct 11 10:41:03.953408 master-1 kubenswrapper[4771]: I1011 10:41:03.953387 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61"} err="failed to get container status \"099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61\": rpc error: code = NotFound desc = could not find container \"099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61\": container with ID starting with 099a8b3dcef0438896afc75fcd82f68fe99e85fb11c77c0389001ba13a5e3c61 not found: ID does not exist" Oct 11 10:41:03.953995 master-1 kubenswrapper[4771]: I1011 10:41:03.953935 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-oauth-apiserver/apiserver-68f4c55ff4-z898b"] Oct 11 10:41:03.974381 master-1 kubenswrapper[4771]: I1011 10:41:03.970511 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:41:04.197101 master-1 kubenswrapper[4771]: I1011 10:41:04.197029 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc"] Oct 11 10:41:04.203878 master-1 kubenswrapper[4771]: W1011 10:41:04.203821 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7647696_42d9_4dd9_bc3b_a4d52a42cf9a.slice/crio-427eb40cfa0882eb2802d3c999b5c0efc782d64ced889beb36b9629f8b535d5d WatchSource:0}: Error finding container 427eb40cfa0882eb2802d3c999b5c0efc782d64ced889beb36b9629f8b535d5d: Status 404 returned error can't find the container with id 427eb40cfa0882eb2802d3c999b5c0efc782d64ced889beb36b9629f8b535d5d Oct 11 10:41:04.450491 master-1 kubenswrapper[4771]: I1011 10:41:04.450435 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" path="/var/lib/kubelet/pods/44bb7164-0bee-4f90-8bf6-a2d73e1f3d40/volumes" Oct 11 10:41:04.451379 master-1 kubenswrapper[4771]: I1011 10:41:04.451331 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm"] Oct 11 10:41:04.893199 master-1 kubenswrapper[4771]: I1011 10:41:04.893126 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" event={"ID":"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b","Type":"ContainerStarted","Data":"b26edef884aff3f6672e1cf391fedcd8f44db05f9d32b8b278db114403a7ea30"} Oct 11 10:41:04.893199 master-1 kubenswrapper[4771]: I1011 10:41:04.893175 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" event={"ID":"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b","Type":"ContainerStarted","Data":"6872f9dd722eb9def4b0c8368459a22a1b55a6810993877af216d31bdec99d96"} Oct 11 10:41:04.894176 master-1 kubenswrapper[4771]: I1011 10:41:04.894136 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" event={"ID":"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a","Type":"ContainerStarted","Data":"427eb40cfa0882eb2802d3c999b5c0efc782d64ced889beb36b9629f8b535d5d"} Oct 11 10:41:05.539492 master-1 kubenswrapper[4771]: I1011 10:41:05.539441 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:41:05.539492 master-1 kubenswrapper[4771]: I1011 10:41:05.539493 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: I1011 10:41:05.547076 4771 patch_prober.go:28] interesting pod/kube-apiserver-master-1 container/kube-apiserver namespace/openshift-kube-apiserver: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]etcd ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [-]poststarthook/rbac/bootstrap-roles failed: reason withheld Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:41:05.547170 master-1 kubenswrapper[4771]: livez check failed Oct 11 10:41:05.548416 master-1 kubenswrapper[4771]: I1011 10:41:05.547224 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:41:07.251499 master-1 kubenswrapper[4771]: I1011 10:41:07.251421 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:07.252046 master-1 kubenswrapper[4771]: I1011 10:41:07.251513 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:07.924041 master-1 kubenswrapper[4771]: I1011 10:41:07.923964 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" event={"ID":"6d66210c-3e1b-4f1f-87b7-6dfe4dd5423b","Type":"ContainerStarted","Data":"90dd61f5266758da211b0e98508c1a1d319122d1e8ed03c42c48e7345067cef4"} Oct 11 10:41:07.924431 master-1 kubenswrapper[4771]: I1011 10:41:07.924165 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:41:07.927480 master-1 kubenswrapper[4771]: I1011 10:41:07.927402 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" event={"ID":"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a","Type":"ContainerStarted","Data":"ee7362ce753dcdd54615e3606ada5492960b78f2a0918c80fdb33aa93c30dbd6"} Oct 11 10:41:07.927943 master-1 kubenswrapper[4771]: I1011 10:41:07.927495 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" event={"ID":"d7647696-42d9-4dd9-bc3b-a4d52a42cf9a","Type":"ContainerStarted","Data":"f86aabc69abdb933f1234a37c50d6fc06722446595529fab70389a6a64dd2e2f"} Oct 11 10:41:07.927943 master-1 kubenswrapper[4771]: I1011 10:41:07.927600 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:41:07.957138 master-1 kubenswrapper[4771]: I1011 10:41:07.957016 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" podStartSLOduration=741.193524109 podStartE2EDuration="12m22.956992471s" podCreationTimestamp="2025-10-11 10:28:45 +0000 UTC" firstStartedPulling="2025-10-11 10:41:04.848863518 +0000 UTC m=+896.823089959" lastFinishedPulling="2025-10-11 10:41:06.61233188 +0000 UTC m=+898.586558321" observedRunningTime="2025-10-11 10:41:07.952262453 +0000 UTC m=+899.926488974" watchObservedRunningTime="2025-10-11 10:41:07.956992471 +0000 UTC m=+899.931218952" Oct 11 10:41:08.242279 master-1 kubenswrapper[4771]: I1011 10:41:08.242166 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:41:08.264442 master-1 kubenswrapper[4771]: I1011 10:41:08.264291 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" podStartSLOduration=740.855158064 podStartE2EDuration="12m23.264257682s" podCreationTimestamp="2025-10-11 10:28:45 +0000 UTC" firstStartedPulling="2025-10-11 10:41:04.207120895 +0000 UTC m=+896.181347356" lastFinishedPulling="2025-10-11 10:41:06.616220523 +0000 UTC m=+898.590446974" observedRunningTime="2025-10-11 10:41:08.004424738 +0000 UTC m=+899.978651239" watchObservedRunningTime="2025-10-11 10:41:08.264257682 +0000 UTC m=+900.238484153" Oct 11 10:41:09.689769 master-1 kubenswrapper[4771]: I1011 10:41:09.689678 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:41:09.737825 master-1 kubenswrapper[4771]: I1011 10:41:09.737696 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:41:09.984792 master-1 kubenswrapper[4771]: I1011 10:41:09.984599 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-1" Oct 11 10:41:10.167379 master-1 kubenswrapper[4771]: I1011 10:41:10.167261 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-console/console-775ff6c4fc-csp4z"] Oct 11 10:41:10.548473 master-1 kubenswrapper[4771]: I1011 10:41:10.548398 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:41:10.553091 master-1 kubenswrapper[4771]: I1011 10:41:10.553060 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:41:10.568924 master-1 kubenswrapper[4771]: I1011 10:41:10.568862 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-oauth-apiserver/apiserver-656768b4df-g4p26"] Oct 11 10:41:10.569686 master-1 kubenswrapper[4771]: E1011 10:41:10.569645 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="fix-audit-permissions" Oct 11 10:41:10.569686 master-1 kubenswrapper[4771]: I1011 10:41:10.569677 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="fix-audit-permissions" Oct 11 10:41:10.569804 master-1 kubenswrapper[4771]: E1011 10:41:10.569715 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" Oct 11 10:41:10.569804 master-1 kubenswrapper[4771]: I1011 10:41:10.569727 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" Oct 11 10:41:10.569953 master-1 kubenswrapper[4771]: I1011 10:41:10.569919 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="44bb7164-0bee-4f90-8bf6-a2d73e1f3d40" containerName="oauth-apiserver" Oct 11 10:41:10.571305 master-1 kubenswrapper[4771]: I1011 10:41:10.571259 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.574311 master-1 kubenswrapper[4771]: I1011 10:41:10.574246 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"audit-1" Oct 11 10:41:10.574782 master-1 kubenswrapper[4771]: I1011 10:41:10.574715 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"trusted-ca-bundle" Oct 11 10:41:10.575115 master-1 kubenswrapper[4771]: I1011 10:41:10.574808 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"encryption-config-1" Oct 11 10:41:10.575115 master-1 kubenswrapper[4771]: I1011 10:41:10.574865 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"etcd-serving-ca" Oct 11 10:41:10.575115 master-1 kubenswrapper[4771]: I1011 10:41:10.575099 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"openshift-service-ca.crt" Oct 11 10:41:10.575115 master-1 kubenswrapper[4771]: I1011 10:41:10.575155 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"etcd-client" Oct 11 10:41:10.575115 master-1 kubenswrapper[4771]: I1011 10:41:10.575285 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"oauth-apiserver-sa-dockercfg-zlnjr" Oct 11 10:41:10.575115 master-1 kubenswrapper[4771]: I1011 10:41:10.575790 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"kube-root-ca.crt" Oct 11 10:41:10.575115 master-1 kubenswrapper[4771]: I1011 10:41:10.576094 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"serving-cert" Oct 11 10:41:10.592702 master-1 kubenswrapper[4771]: I1011 10:41:10.592631 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-trusted-ca-bundle\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.592887 master-1 kubenswrapper[4771]: I1011 10:41:10.592726 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-audit-policies\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.592887 master-1 kubenswrapper[4771]: I1011 10:41:10.592781 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjn45\" (UniqueName: \"kubernetes.io/projected/4c0cf305-ba21-45c0-a092-05214809da68-kube-api-access-rjn45\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.592887 master-1 kubenswrapper[4771]: I1011 10:41:10.592818 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/4c0cf305-ba21-45c0-a092-05214809da68-audit-dir\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.592887 master-1 kubenswrapper[4771]: I1011 10:41:10.592885 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-encryption-config\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.593138 master-1 kubenswrapper[4771]: I1011 10:41:10.592914 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-serving-cert\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.593138 master-1 kubenswrapper[4771]: I1011 10:41:10.592950 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-etcd-serving-ca\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.593138 master-1 kubenswrapper[4771]: I1011 10:41:10.592980 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-etcd-client\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.596571 master-1 kubenswrapper[4771]: I1011 10:41:10.596506 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-656768b4df-g4p26"] Oct 11 10:41:10.694759 master-1 kubenswrapper[4771]: I1011 10:41:10.694558 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-etcd-serving-ca\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.694759 master-1 kubenswrapper[4771]: I1011 10:41:10.694667 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-etcd-client\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.694759 master-1 kubenswrapper[4771]: I1011 10:41:10.694727 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-trusted-ca-bundle\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.695514 master-1 kubenswrapper[4771]: I1011 10:41:10.694775 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-audit-policies\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.695514 master-1 kubenswrapper[4771]: I1011 10:41:10.694821 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-rjn45\" (UniqueName: \"kubernetes.io/projected/4c0cf305-ba21-45c0-a092-05214809da68-kube-api-access-rjn45\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.695514 master-1 kubenswrapper[4771]: I1011 10:41:10.694854 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/4c0cf305-ba21-45c0-a092-05214809da68-audit-dir\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.695514 master-1 kubenswrapper[4771]: I1011 10:41:10.694942 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-encryption-config\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.695514 master-1 kubenswrapper[4771]: I1011 10:41:10.694982 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-serving-cert\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.695514 master-1 kubenswrapper[4771]: I1011 10:41:10.695076 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/4c0cf305-ba21-45c0-a092-05214809da68-audit-dir\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.695750 master-1 kubenswrapper[4771]: I1011 10:41:10.695615 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-etcd-serving-ca\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.695796 master-1 kubenswrapper[4771]: I1011 10:41:10.695732 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-trusted-ca-bundle\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.696108 master-1 kubenswrapper[4771]: I1011 10:41:10.696044 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-audit-policies\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.699305 master-1 kubenswrapper[4771]: I1011 10:41:10.699259 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-serving-cert\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.700280 master-1 kubenswrapper[4771]: I1011 10:41:10.700222 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-etcd-client\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.706812 master-1 kubenswrapper[4771]: I1011 10:41:10.706760 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-encryption-config\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.744203 master-1 kubenswrapper[4771]: I1011 10:41:10.744129 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjn45\" (UniqueName: \"kubernetes.io/projected/4c0cf305-ba21-45c0-a092-05214809da68-kube-api-access-rjn45\") pod \"apiserver-656768b4df-g4p26\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:10.899203 master-1 kubenswrapper[4771]: I1011 10:41:10.898898 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:11.453792 master-1 kubenswrapper[4771]: I1011 10:41:11.453703 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-656768b4df-g4p26"] Oct 11 10:41:11.472412 master-1 kubenswrapper[4771]: W1011 10:41:11.472334 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0cf305_ba21_45c0_a092_05214809da68.slice/crio-9a8773a82720172e1c708c6b8b379786c06f2193ace376125888c909cd115b04 WatchSource:0}: Error finding container 9a8773a82720172e1c708c6b8b379786c06f2193ace376125888c909cd115b04: Status 404 returned error can't find the container with id 9a8773a82720172e1c708c6b8b379786c06f2193ace376125888c909cd115b04 Oct 11 10:41:11.973623 master-1 kubenswrapper[4771]: I1011 10:41:11.973456 4771 generic.go:334] "Generic (PLEG): container finished" podID="4c0cf305-ba21-45c0-a092-05214809da68" containerID="0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22" exitCode=0 Oct 11 10:41:11.973623 master-1 kubenswrapper[4771]: I1011 10:41:11.973564 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" event={"ID":"4c0cf305-ba21-45c0-a092-05214809da68","Type":"ContainerDied","Data":"0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22"} Oct 11 10:41:11.974149 master-1 kubenswrapper[4771]: I1011 10:41:11.973641 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" event={"ID":"4c0cf305-ba21-45c0-a092-05214809da68","Type":"ContainerStarted","Data":"9a8773a82720172e1c708c6b8b379786c06f2193ace376125888c909cd115b04"} Oct 11 10:41:12.251411 master-1 kubenswrapper[4771]: I1011 10:41:12.251314 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:12.251595 master-1 kubenswrapper[4771]: I1011 10:41:12.251423 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:12.983092 master-1 kubenswrapper[4771]: I1011 10:41:12.982960 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" event={"ID":"4c0cf305-ba21-45c0-a092-05214809da68","Type":"ContainerStarted","Data":"0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948"} Oct 11 10:41:13.033975 master-1 kubenswrapper[4771]: I1011 10:41:13.033865 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podStartSLOduration=62.033836215 podStartE2EDuration="1m2.033836215s" podCreationTimestamp="2025-10-11 10:40:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:41:13.028280354 +0000 UTC m=+905.002506815" watchObservedRunningTime="2025-10-11 10:41:13.033836215 +0000 UTC m=+905.008062676" Oct 11 10:41:13.669731 master-1 kubenswrapper[4771]: I1011 10:41:13.669652 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operator-controller/operator-controller-controller-manager-668cb7cdc8-bqdlc" Oct 11 10:41:13.976451 master-1 kubenswrapper[4771]: I1011 10:41:13.976272 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-catalogd/catalogd-controller-manager-596f9d8bbf-tpzsm" Oct 11 10:41:15.900510 master-1 kubenswrapper[4771]: I1011 10:41:15.900423 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:15.901389 master-1 kubenswrapper[4771]: I1011 10:41:15.900542 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:15.912103 master-1 kubenswrapper[4771]: I1011 10:41:15.912038 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:16.014836 master-1 kubenswrapper[4771]: I1011 10:41:16.014744 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:17.251262 master-1 kubenswrapper[4771]: I1011 10:41:17.251130 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:17.251262 master-1 kubenswrapper[4771]: I1011 10:41:17.251234 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:20.545126 master-1 kubenswrapper[4771]: I1011 10:41:20.545070 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:41:22.250825 master-1 kubenswrapper[4771]: I1011 10:41:22.250755 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:22.251767 master-1 kubenswrapper[4771]: I1011 10:41:22.251662 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:27.250987 master-1 kubenswrapper[4771]: I1011 10:41:27.250902 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:27.250987 master-1 kubenswrapper[4771]: I1011 10:41:27.250981 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:32.251007 master-1 kubenswrapper[4771]: I1011 10:41:32.250908 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:32.251848 master-1 kubenswrapper[4771]: I1011 10:41:32.251031 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:35.203924 master-1 kubenswrapper[4771]: I1011 10:41:35.203830 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-console/console-775ff6c4fc-csp4z" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" containerID="cri-o://10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5" gracePeriod=15 Oct 11 10:41:35.768142 master-1 kubenswrapper[4771]: I1011 10:41:35.768109 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-775ff6c4fc-csp4z_de7aa64b-afab-4b3a-b56d-81c324e7a8cb/console/0.log" Oct 11 10:41:35.768257 master-1 kubenswrapper[4771]: I1011 10:41:35.768194 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:41:35.818769 master-1 kubenswrapper[4771]: I1011 10:41:35.815158 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b846b7bb4-xmv6l"] Oct 11 10:41:35.818769 master-1 kubenswrapper[4771]: E1011 10:41:35.815569 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" Oct 11 10:41:35.818769 master-1 kubenswrapper[4771]: I1011 10:41:35.815594 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" Oct 11 10:41:35.818769 master-1 kubenswrapper[4771]: I1011 10:41:35.815809 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerName="console" Oct 11 10:41:35.818769 master-1 kubenswrapper[4771]: I1011 10:41:35.816534 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:35.843700 master-1 kubenswrapper[4771]: I1011 10:41:35.839562 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b846b7bb4-xmv6l"] Oct 11 10:41:35.900856 master-1 kubenswrapper[4771]: I1011 10:41:35.900777 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmzlm\" (UniqueName: \"kubernetes.io/projected/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-kube-api-access-mmzlm\") pod \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " Oct 11 10:41:35.901107 master-1 kubenswrapper[4771]: I1011 10:41:35.900942 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-trusted-ca-bundle\") pod \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " Oct 11 10:41:35.901107 master-1 kubenswrapper[4771]: I1011 10:41:35.900984 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-config\") pod \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " Oct 11 10:41:35.901107 master-1 kubenswrapper[4771]: I1011 10:41:35.901011 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-oauth-serving-cert\") pod \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " Oct 11 10:41:35.901107 master-1 kubenswrapper[4771]: I1011 10:41:35.901060 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-oauth-config\") pod \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " Oct 11 10:41:35.901107 master-1 kubenswrapper[4771]: I1011 10:41:35.901090 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-service-ca\") pod \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " Oct 11 10:41:35.901580 master-1 kubenswrapper[4771]: I1011 10:41:35.901120 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-serving-cert\") pod \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\" (UID: \"de7aa64b-afab-4b3a-b56d-81c324e7a8cb\") " Oct 11 10:41:35.901580 master-1 kubenswrapper[4771]: I1011 10:41:35.901326 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-service-ca\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:35.901580 master-1 kubenswrapper[4771]: I1011 10:41:35.901416 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-oauth-serving-cert\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:35.901580 master-1 kubenswrapper[4771]: I1011 10:41:35.901490 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpp7h\" (UniqueName: \"kubernetes.io/projected/a65b0165-5747-48c9-9179-86f19861dd68-kube-api-access-qpp7h\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:35.901580 master-1 kubenswrapper[4771]: I1011 10:41:35.901523 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-console-config\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:35.901580 master-1 kubenswrapper[4771]: I1011 10:41:35.901557 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-serving-cert\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:35.903190 master-1 kubenswrapper[4771]: I1011 10:41:35.901596 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-oauth-config\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:35.903190 master-1 kubenswrapper[4771]: I1011 10:41:35.901636 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-trusted-ca-bundle\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:35.903190 master-1 kubenswrapper[4771]: I1011 10:41:35.902493 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "de7aa64b-afab-4b3a-b56d-81c324e7a8cb" (UID: "de7aa64b-afab-4b3a-b56d-81c324e7a8cb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:41:35.903190 master-1 kubenswrapper[4771]: I1011 10:41:35.902798 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-service-ca" (OuterVolumeSpecName: "service-ca") pod "de7aa64b-afab-4b3a-b56d-81c324e7a8cb" (UID: "de7aa64b-afab-4b3a-b56d-81c324e7a8cb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:41:35.903190 master-1 kubenswrapper[4771]: I1011 10:41:35.902812 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "de7aa64b-afab-4b3a-b56d-81c324e7a8cb" (UID: "de7aa64b-afab-4b3a-b56d-81c324e7a8cb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:41:35.904256 master-1 kubenswrapper[4771]: I1011 10:41:35.904115 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "de7aa64b-afab-4b3a-b56d-81c324e7a8cb" (UID: "de7aa64b-afab-4b3a-b56d-81c324e7a8cb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:41:35.906215 master-1 kubenswrapper[4771]: I1011 10:41:35.906153 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-kube-api-access-mmzlm" (OuterVolumeSpecName: "kube-api-access-mmzlm") pod "de7aa64b-afab-4b3a-b56d-81c324e7a8cb" (UID: "de7aa64b-afab-4b3a-b56d-81c324e7a8cb"). InnerVolumeSpecName "kube-api-access-mmzlm". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:41:35.906698 master-1 kubenswrapper[4771]: I1011 10:41:35.906604 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-config" (OuterVolumeSpecName: "console-config") pod "de7aa64b-afab-4b3a-b56d-81c324e7a8cb" (UID: "de7aa64b-afab-4b3a-b56d-81c324e7a8cb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:41:35.906879 master-1 kubenswrapper[4771]: I1011 10:41:35.906709 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "de7aa64b-afab-4b3a-b56d-81c324e7a8cb" (UID: "de7aa64b-afab-4b3a-b56d-81c324e7a8cb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003567 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-qpp7h\" (UniqueName: \"kubernetes.io/projected/a65b0165-5747-48c9-9179-86f19861dd68-kube-api-access-qpp7h\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003650 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-console-config\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003703 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-serving-cert\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003726 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-oauth-config\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003751 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-trusted-ca-bundle\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003782 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-service-ca\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003810 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-oauth-serving-cert\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003875 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003891 4771 reconciler_common.go:293] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.003901 4771 reconciler_common.go:293] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-oauth-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.004056 4771 reconciler_common.go:293] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-oauth-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.005987 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-oauth-serving-cert\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.006104 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-service-ca\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.006142 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-trusted-ca-bundle\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.006153 4771 reconciler_common.go:293] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-service-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.006232 4771 reconciler_common.go:293] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-console-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.006259 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-mmzlm\" (UniqueName: \"kubernetes.io/projected/de7aa64b-afab-4b3a-b56d-81c324e7a8cb-kube-api-access-mmzlm\") on node \"master-1\" DevicePath \"\"" Oct 11 10:41:36.008124 master-1 kubenswrapper[4771]: I1011 10:41:36.006577 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-console-config\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.010674 master-1 kubenswrapper[4771]: I1011 10:41:36.010615 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-serving-cert\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.010948 master-1 kubenswrapper[4771]: I1011 10:41:36.010875 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-oauth-config\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.040817 master-1 kubenswrapper[4771]: I1011 10:41:36.040713 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpp7h\" (UniqueName: \"kubernetes.io/projected/a65b0165-5747-48c9-9179-86f19861dd68-kube-api-access-qpp7h\") pod \"console-5b846b7bb4-xmv6l\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.145450 master-1 kubenswrapper[4771]: I1011 10:41:36.145335 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:36.187872 master-1 kubenswrapper[4771]: I1011 10:41:36.187799 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-775ff6c4fc-csp4z_de7aa64b-afab-4b3a-b56d-81c324e7a8cb/console/0.log" Oct 11 10:41:36.188041 master-1 kubenswrapper[4771]: I1011 10:41:36.187887 4771 generic.go:334] "Generic (PLEG): container finished" podID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" containerID="10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5" exitCode=2 Oct 11 10:41:36.188041 master-1 kubenswrapper[4771]: I1011 10:41:36.187934 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775ff6c4fc-csp4z" event={"ID":"de7aa64b-afab-4b3a-b56d-81c324e7a8cb","Type":"ContainerDied","Data":"10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5"} Oct 11 10:41:36.188041 master-1 kubenswrapper[4771]: I1011 10:41:36.187984 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775ff6c4fc-csp4z" event={"ID":"de7aa64b-afab-4b3a-b56d-81c324e7a8cb","Type":"ContainerDied","Data":"dbb5133e318020821233bd4743645ca9f974f8d4348733f58f43c17203dfa102"} Oct 11 10:41:36.188041 master-1 kubenswrapper[4771]: I1011 10:41:36.187997 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775ff6c4fc-csp4z" Oct 11 10:41:36.188491 master-1 kubenswrapper[4771]: I1011 10:41:36.188015 4771 scope.go:117] "RemoveContainer" containerID="10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5" Oct 11 10:41:36.220411 master-1 kubenswrapper[4771]: I1011 10:41:36.219768 4771 scope.go:117] "RemoveContainer" containerID="10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5" Oct 11 10:41:36.221064 master-1 kubenswrapper[4771]: E1011 10:41:36.220969 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5\": container with ID starting with 10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5 not found: ID does not exist" containerID="10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5" Oct 11 10:41:36.221064 master-1 kubenswrapper[4771]: I1011 10:41:36.221004 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5"} err="failed to get container status \"10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5\": rpc error: code = NotFound desc = could not find container \"10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5\": container with ID starting with 10775f41088b2fc502cb2185cba3870f7995bc3ef1e5d846bebb3b393b7337f5 not found: ID does not exist" Oct 11 10:41:36.248066 master-1 kubenswrapper[4771]: I1011 10:41:36.248004 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-console/console-775ff6c4fc-csp4z"] Oct 11 10:41:36.261260 master-1 kubenswrapper[4771]: I1011 10:41:36.261148 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-775ff6c4fc-csp4z"] Oct 11 10:41:36.458576 master-1 kubenswrapper[4771]: I1011 10:41:36.458494 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de7aa64b-afab-4b3a-b56d-81c324e7a8cb" path="/var/lib/kubelet/pods/de7aa64b-afab-4b3a-b56d-81c324e7a8cb/volumes" Oct 11 10:41:36.659720 master-1 kubenswrapper[4771]: I1011 10:41:36.659674 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b846b7bb4-xmv6l"] Oct 11 10:41:36.670102 master-1 kubenswrapper[4771]: W1011 10:41:36.670034 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65b0165_5747_48c9_9179_86f19861dd68.slice/crio-2e652727629cf31e7de5014abdf61de5e97f13fd0cbfe170fa06452ef6ed0070 WatchSource:0}: Error finding container 2e652727629cf31e7de5014abdf61de5e97f13fd0cbfe170fa06452ef6ed0070: Status 404 returned error can't find the container with id 2e652727629cf31e7de5014abdf61de5e97f13fd0cbfe170fa06452ef6ed0070 Oct 11 10:41:37.199468 master-1 kubenswrapper[4771]: I1011 10:41:37.199328 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b846b7bb4-xmv6l" event={"ID":"a65b0165-5747-48c9-9179-86f19861dd68","Type":"ContainerStarted","Data":"6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6"} Oct 11 10:41:37.200080 master-1 kubenswrapper[4771]: I1011 10:41:37.199491 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b846b7bb4-xmv6l" event={"ID":"a65b0165-5747-48c9-9179-86f19861dd68","Type":"ContainerStarted","Data":"2e652727629cf31e7de5014abdf61de5e97f13fd0cbfe170fa06452ef6ed0070"} Oct 11 10:41:37.252459 master-1 kubenswrapper[4771]: I1011 10:41:37.252389 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:37.253333 master-1 kubenswrapper[4771]: I1011 10:41:37.253279 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:42.252972 master-1 kubenswrapper[4771]: I1011 10:41:42.252855 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:42.252972 master-1 kubenswrapper[4771]: I1011 10:41:42.252946 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:42.581086 master-1 kubenswrapper[4771]: I1011 10:41:42.580960 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5b846b7bb4-xmv6l" podStartSLOduration=21.580927478 podStartE2EDuration="21.580927478s" podCreationTimestamp="2025-10-11 10:41:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:41:37.229689597 +0000 UTC m=+929.203916128" watchObservedRunningTime="2025-10-11 10:41:42.580927478 +0000 UTC m=+934.555153929" Oct 11 10:41:42.581884 master-1 kubenswrapper[4771]: I1011 10:41:42.581846 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-oauth-apiserver/apiserver-656768b4df-g4p26"] Oct 11 10:41:42.582313 master-1 kubenswrapper[4771]: I1011 10:41:42.582262 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" containerID="cri-o://0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948" gracePeriod=120 Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: I1011 10:41:45.908466 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:41:45.908586 master-1 kubenswrapper[4771]: I1011 10:41:45.908577 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:41:46.146172 master-1 kubenswrapper[4771]: I1011 10:41:46.146078 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:46.146172 master-1 kubenswrapper[4771]: I1011 10:41:46.146162 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:46.154009 master-1 kubenswrapper[4771]: I1011 10:41:46.153954 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:46.272194 master-1 kubenswrapper[4771]: I1011 10:41:46.272011 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:41:47.250932 master-1 kubenswrapper[4771]: I1011 10:41:47.250813 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:47.250932 master-1 kubenswrapper[4771]: I1011 10:41:47.250903 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: I1011 10:41:50.906741 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:41:50.906815 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:41:50.908122 master-1 kubenswrapper[4771]: I1011 10:41:50.906823 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:41:52.251168 master-1 kubenswrapper[4771]: I1011 10:41:52.251069 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:52.251168 master-1 kubenswrapper[4771]: I1011 10:41:52.251122 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: I1011 10:41:55.905798 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:41:55.905876 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:41:55.907545 master-1 kubenswrapper[4771]: I1011 10:41:55.907487 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:41:55.907875 master-1 kubenswrapper[4771]: I1011 10:41:55.907843 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:41:57.251651 master-1 kubenswrapper[4771]: I1011 10:41:57.251572 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:41:57.252341 master-1 kubenswrapper[4771]: I1011 10:41:57.251653 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: I1011 10:42:00.908750 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:42:00.908849 master-1 kubenswrapper[4771]: I1011 10:42:00.908847 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:42:01.272264 master-1 kubenswrapper[4771]: I1011 10:42:01.270571 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-etcd/installer-9-master-1"] Oct 11 10:42:01.274649 master-1 kubenswrapper[4771]: I1011 10:42:01.274611 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.279955 master-1 kubenswrapper[4771]: I1011 10:42:01.279340 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-etcd"/"installer-sa-dockercfg-xbqxb" Oct 11 10:42:01.285206 master-1 kubenswrapper[4771]: I1011 10:42:01.285155 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/installer-9-master-1"] Oct 11 10:42:01.318143 master-1 kubenswrapper[4771]: I1011 10:42:01.318069 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-kubelet-dir\") pod \"installer-9-master-1\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.318633 master-1 kubenswrapper[4771]: I1011 10:42:01.318589 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/1da01b89-3c1e-4f11-bcc6-65a56654021f-kube-api-access\") pod \"installer-9-master-1\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.318898 master-1 kubenswrapper[4771]: I1011 10:42:01.318864 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-var-lock\") pod \"installer-9-master-1\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.420961 master-1 kubenswrapper[4771]: I1011 10:42:01.420915 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-kubelet-dir\") pod \"installer-9-master-1\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.421204 master-1 kubenswrapper[4771]: I1011 10:42:01.421186 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/1da01b89-3c1e-4f11-bcc6-65a56654021f-kube-api-access\") pod \"installer-9-master-1\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.421329 master-1 kubenswrapper[4771]: I1011 10:42:01.421316 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-var-lock\") pod \"installer-9-master-1\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.421523 master-1 kubenswrapper[4771]: I1011 10:42:01.421483 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-kubelet-dir\") pod \"installer-9-master-1\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.421582 master-1 kubenswrapper[4771]: I1011 10:42:01.421508 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-var-lock\") pod \"installer-9-master-1\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.446042 master-1 kubenswrapper[4771]: I1011 10:42:01.445976 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/1da01b89-3c1e-4f11-bcc6-65a56654021f-kube-api-access\") pod \"installer-9-master-1\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:01.605348 master-1 kubenswrapper[4771]: I1011 10:42:01.605272 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:02.012551 master-1 kubenswrapper[4771]: I1011 10:42:02.012413 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/installer-9-master-1"] Oct 11 10:42:02.251641 master-1 kubenswrapper[4771]: I1011 10:42:02.251587 4771 patch_prober.go:28] interesting pod/apiserver-7845cf54d8-g8x5z container/openshift-apiserver namespace/openshift-apiserver: Readiness probe status=failure output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" start-of-body= Oct 11 10:42:02.251794 master-1 kubenswrapper[4771]: I1011 10:42:02.251653 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" probeResult="failure" output="Get \"https://10.129.0.53:8443/readyz?exclude=etcd&exclude=etcd-readiness\": dial tcp 10.129.0.53:8443: connect: connection refused" Oct 11 10:42:02.383625 master-1 kubenswrapper[4771]: I1011 10:42:02.381813 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-9-master-1" event={"ID":"1da01b89-3c1e-4f11-bcc6-65a56654021f","Type":"ContainerStarted","Data":"9692635e40b2a711a263503ff5795f641f5480d42c4c64a94f91d9bd4aff98f6"} Oct 11 10:42:03.394345 master-1 kubenswrapper[4771]: I1011 10:42:03.393969 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-9-master-1" event={"ID":"1da01b89-3c1e-4f11-bcc6-65a56654021f","Type":"ContainerStarted","Data":"1a1e8546ece3b9b09f96eb38ce98e4e2f7676e9d011955a8c3b8f572088b6cdb"} Oct 11 10:42:03.422264 master-1 kubenswrapper[4771]: I1011 10:42:03.422192 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-etcd/installer-9-master-1" podStartSLOduration=2.422175243 podStartE2EDuration="2.422175243s" podCreationTimestamp="2025-10-11 10:42:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:42:03.415331635 +0000 UTC m=+955.389558106" watchObservedRunningTime="2025-10-11 10:42:03.422175243 +0000 UTC m=+955.396401694" Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: I1011 10:42:05.905126 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:42:05.905218 master-1 kubenswrapper[4771]: I1011 10:42:05.905216 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:42:06.417952 master-1 kubenswrapper[4771]: I1011 10:42:06.417871 4771 generic.go:334] "Generic (PLEG): container finished" podID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerID="2ccd5ea4ca8c2b32e04ef7419d2c1c1ac0971dd1b18e1a37cd16058b70e5a98c" exitCode=0 Oct 11 10:42:06.418170 master-1 kubenswrapper[4771]: I1011 10:42:06.417941 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" event={"ID":"a2bf529d-094c-4406-8ce6-890cf8c0b840","Type":"ContainerDied","Data":"2ccd5ea4ca8c2b32e04ef7419d2c1c1ac0971dd1b18e1a37cd16058b70e5a98c"} Oct 11 10:42:07.119064 master-1 kubenswrapper[4771]: I1011 10:42:07.119000 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:42:07.208838 master-1 kubenswrapper[4771]: I1011 10:42:07.208728 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-apiserver/apiserver-8865994fd-g2fnh"] Oct 11 10:42:07.209327 master-1 kubenswrapper[4771]: E1011 10:42:07.209062 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver-check-endpoints" Oct 11 10:42:07.209327 master-1 kubenswrapper[4771]: I1011 10:42:07.209086 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver-check-endpoints" Oct 11 10:42:07.209327 master-1 kubenswrapper[4771]: E1011 10:42:07.209120 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="fix-audit-permissions" Oct 11 10:42:07.209327 master-1 kubenswrapper[4771]: I1011 10:42:07.209134 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="fix-audit-permissions" Oct 11 10:42:07.209327 master-1 kubenswrapper[4771]: E1011 10:42:07.209156 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" Oct 11 10:42:07.209327 master-1 kubenswrapper[4771]: I1011 10:42:07.209172 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" Oct 11 10:42:07.211334 master-1 kubenswrapper[4771]: I1011 10:42:07.209337 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-client\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.211334 master-1 kubenswrapper[4771]: I1011 10:42:07.209403 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver" Oct 11 10:42:07.211334 master-1 kubenswrapper[4771]: I1011 10:42:07.209619 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-config\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.211334 master-1 kubenswrapper[4771]: I1011 10:42:07.209660 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" containerName="openshift-apiserver-check-endpoints" Oct 11 10:42:07.211334 master-1 kubenswrapper[4771]: I1011 10:42:07.210331 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-config" (OuterVolumeSpecName: "config") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:42:07.211334 master-1 kubenswrapper[4771]: I1011 10:42:07.210514 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-serving-cert\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.211334 master-1 kubenswrapper[4771]: I1011 10:42:07.210647 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-image-import-ca\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.211334 master-1 kubenswrapper[4771]: I1011 10:42:07.210742 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-encryption-config\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.211334 master-1 kubenswrapper[4771]: I1011 10:42:07.211140 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.211901 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-image-import-ca" (OuterVolumeSpecName: "image-import-ca") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "image-import-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.212455 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.212519 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-trusted-ca-bundle\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.212656 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.212708 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-node-pullsecrets\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.212746 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit-dir\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.212785 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-serving-ca\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.212815 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b78cw\" (UniqueName: \"kubernetes.io/projected/a2bf529d-094c-4406-8ce6-890cf8c0b840-kube-api-access-b78cw\") pod \"a2bf529d-094c-4406-8ce6-890cf8c0b840\" (UID: \"a2bf529d-094c-4406-8ce6-890cf8c0b840\") " Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.212806 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-node-pullsecrets" (OuterVolumeSpecName: "node-pullsecrets") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "node-pullsecrets". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.212828 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.213279 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit" (OuterVolumeSpecName: "audit") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "audit". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.213447 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-serving-ca" (OuterVolumeSpecName: "etcd-serving-ca") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "etcd-serving-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.213680 4771 reconciler_common.go:293] "Volume detached for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-image-import-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.213698 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.213711 4771 reconciler_common.go:293] "Volume detached for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.213721 4771 reconciler_common.go:293] "Volume detached for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-node-pullsecrets\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.213730 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/a2bf529d-094c-4406-8ce6-890cf8c0b840-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.213741 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-serving-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.213939 master-1 kubenswrapper[4771]: I1011 10:42:07.213750 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2bf529d-094c-4406-8ce6-890cf8c0b840-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.215618 master-1 kubenswrapper[4771]: I1011 10:42:07.214402 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-apiserver"/"openshift-apiserver-sa-dockercfg-lntq9" Oct 11 10:42:07.215618 master-1 kubenswrapper[4771]: I1011 10:42:07.215477 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-client" (OuterVolumeSpecName: "etcd-client") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "etcd-client". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:42:07.217591 master-1 kubenswrapper[4771]: I1011 10:42:07.217316 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bf529d-094c-4406-8ce6-890cf8c0b840-kube-api-access-b78cw" (OuterVolumeSpecName: "kube-api-access-b78cw") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "kube-api-access-b78cw". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:42:07.219938 master-1 kubenswrapper[4771]: I1011 10:42:07.218832 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:42:07.223738 master-1 kubenswrapper[4771]: I1011 10:42:07.223670 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-apiserver/apiserver-8865994fd-g2fnh"] Oct 11 10:42:07.226063 master-1 kubenswrapper[4771]: I1011 10:42:07.225923 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-encryption-config" (OuterVolumeSpecName: "encryption-config") pod "a2bf529d-094c-4406-8ce6-890cf8c0b840" (UID: "a2bf529d-094c-4406-8ce6-890cf8c0b840"). InnerVolumeSpecName "encryption-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:42:07.315183 master-1 kubenswrapper[4771]: I1011 10:42:07.315133 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-audit-dir\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.315517 master-1 kubenswrapper[4771]: I1011 10:42:07.315495 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-serving-cert\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.315664 master-1 kubenswrapper[4771]: I1011 10:42:07.315642 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-trusted-ca-bundle\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.316453 master-1 kubenswrapper[4771]: I1011 10:42:07.316434 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-etcd-serving-ca\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.316615 master-1 kubenswrapper[4771]: I1011 10:42:07.316597 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-encryption-config\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.316837 master-1 kubenswrapper[4771]: I1011 10:42:07.316816 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-image-import-ca\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.316969 master-1 kubenswrapper[4771]: I1011 10:42:07.316949 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-etcd-client\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.317113 master-1 kubenswrapper[4771]: I1011 10:42:07.317093 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-node-pullsecrets\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.317252 master-1 kubenswrapper[4771]: I1011 10:42:07.317236 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-audit\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.317390 master-1 kubenswrapper[4771]: I1011 10:42:07.317372 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqwt\" (UniqueName: \"kubernetes.io/projected/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-kube-api-access-nzqwt\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.317520 master-1 kubenswrapper[4771]: I1011 10:42:07.317501 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-config\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.317675 master-1 kubenswrapper[4771]: I1011 10:42:07.317641 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.318107 master-1 kubenswrapper[4771]: I1011 10:42:07.318093 4771 reconciler_common.go:293] "Volume detached for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-encryption-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.318256 master-1 kubenswrapper[4771]: I1011 10:42:07.318240 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-b78cw\" (UniqueName: \"kubernetes.io/projected/a2bf529d-094c-4406-8ce6-890cf8c0b840-kube-api-access-b78cw\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.318399 master-1 kubenswrapper[4771]: I1011 10:42:07.318386 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/a2bf529d-094c-4406-8ce6-890cf8c0b840-etcd-client\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:07.420543 master-1 kubenswrapper[4771]: I1011 10:42:07.419314 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-audit\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420543 master-1 kubenswrapper[4771]: I1011 10:42:07.420514 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-audit\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420783 master-1 kubenswrapper[4771]: I1011 10:42:07.420538 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqwt\" (UniqueName: \"kubernetes.io/projected/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-kube-api-access-nzqwt\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420783 master-1 kubenswrapper[4771]: I1011 10:42:07.420588 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-config\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420783 master-1 kubenswrapper[4771]: I1011 10:42:07.420648 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-audit-dir\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420783 master-1 kubenswrapper[4771]: I1011 10:42:07.420703 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-serving-cert\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420783 master-1 kubenswrapper[4771]: I1011 10:42:07.420751 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-trusted-ca-bundle\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420994 master-1 kubenswrapper[4771]: I1011 10:42:07.420792 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-etcd-serving-ca\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420994 master-1 kubenswrapper[4771]: I1011 10:42:07.420833 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-encryption-config\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420994 master-1 kubenswrapper[4771]: I1011 10:42:07.420913 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-etcd-client\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.420994 master-1 kubenswrapper[4771]: I1011 10:42:07.420937 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-image-import-ca\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.421157 master-1 kubenswrapper[4771]: I1011 10:42:07.420995 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-node-pullsecrets\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.421157 master-1 kubenswrapper[4771]: I1011 10:42:07.421128 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-pullsecrets\" (UniqueName: \"kubernetes.io/host-path/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-node-pullsecrets\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.421595 master-1 kubenswrapper[4771]: I1011 10:42:07.421563 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-config\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.421813 master-1 kubenswrapper[4771]: I1011 10:42:07.421777 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-etcd-serving-ca\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.421874 master-1 kubenswrapper[4771]: I1011 10:42:07.421811 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-audit-dir\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.422267 master-1 kubenswrapper[4771]: I1011 10:42:07.422238 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"image-import-ca\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-image-import-ca\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.423677 master-1 kubenswrapper[4771]: I1011 10:42:07.423619 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-trusted-ca-bundle\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.425001 master-1 kubenswrapper[4771]: I1011 10:42:07.424960 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-encryption-config\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.427825 master-1 kubenswrapper[4771]: I1011 10:42:07.427786 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-etcd-client\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.428147 master-1 kubenswrapper[4771]: I1011 10:42:07.428097 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" event={"ID":"a2bf529d-094c-4406-8ce6-890cf8c0b840","Type":"ContainerDied","Data":"e398827cf779d365dfc4e6c2443dd2f776caa9a8ba75c41d00aafc513ef28957"} Oct 11 10:42:07.428214 master-1 kubenswrapper[4771]: I1011 10:42:07.428163 4771 scope.go:117] "RemoveContainer" containerID="a0772db7a40ce6f228f65f235a6668a5f2f1781a4f227000cf9ad01206d856f2" Oct 11 10:42:07.428214 master-1 kubenswrapper[4771]: I1011 10:42:07.428199 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-serving-cert\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.428404 master-1 kubenswrapper[4771]: I1011 10:42:07.428348 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-7845cf54d8-g8x5z" Oct 11 10:42:07.438716 master-1 kubenswrapper[4771]: I1011 10:42:07.438687 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqwt\" (UniqueName: \"kubernetes.io/projected/6c9318b1-5be5-4254-8eb6-7cf411c02eb8-kube-api-access-nzqwt\") pod \"apiserver-8865994fd-g2fnh\" (UID: \"6c9318b1-5be5-4254-8eb6-7cf411c02eb8\") " pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:07.486254 master-1 kubenswrapper[4771]: I1011 10:42:07.486216 4771 scope.go:117] "RemoveContainer" containerID="2ccd5ea4ca8c2b32e04ef7419d2c1c1ac0971dd1b18e1a37cd16058b70e5a98c" Oct 11 10:42:07.501774 master-1 kubenswrapper[4771]: I1011 10:42:07.501742 4771 scope.go:117] "RemoveContainer" containerID="5a44ec551f4491e724d147c13cc98b993a3968bac1f8f715ba1d91a8129c8004" Oct 11 10:42:07.505209 master-1 kubenswrapper[4771]: I1011 10:42:07.505183 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-apiserver/apiserver-7845cf54d8-g8x5z"] Oct 11 10:42:07.513677 master-1 kubenswrapper[4771]: I1011 10:42:07.513640 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-apiserver/apiserver-7845cf54d8-g8x5z"] Oct 11 10:42:07.550214 master-1 kubenswrapper[4771]: I1011 10:42:07.550171 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:08.016515 master-1 kubenswrapper[4771]: I1011 10:42:08.015912 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-apiserver/apiserver-8865994fd-g2fnh"] Oct 11 10:42:08.025995 master-1 kubenswrapper[4771]: W1011 10:42:08.025945 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9318b1_5be5_4254_8eb6_7cf411c02eb8.slice/crio-ac98991a19fe40f0e85bff03a611c5b88f0d91835e7f5d79eaa631518389ccd6 WatchSource:0}: Error finding container ac98991a19fe40f0e85bff03a611c5b88f0d91835e7f5d79eaa631518389ccd6: Status 404 returned error can't find the container with id ac98991a19fe40f0e85bff03a611c5b88f0d91835e7f5d79eaa631518389ccd6 Oct 11 10:42:08.437642 master-1 kubenswrapper[4771]: I1011 10:42:08.437529 4771 generic.go:334] "Generic (PLEG): container finished" podID="6c9318b1-5be5-4254-8eb6-7cf411c02eb8" containerID="f8c9b4d02496959e8fb124becb7d2c60f0137dcae9fa68835aac05f51b1df52b" exitCode=0 Oct 11 10:42:08.453452 master-1 kubenswrapper[4771]: I1011 10:42:08.453387 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2bf529d-094c-4406-8ce6-890cf8c0b840" path="/var/lib/kubelet/pods/a2bf529d-094c-4406-8ce6-890cf8c0b840/volumes" Oct 11 10:42:08.454721 master-1 kubenswrapper[4771]: I1011 10:42:08.454655 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" event={"ID":"6c9318b1-5be5-4254-8eb6-7cf411c02eb8","Type":"ContainerDied","Data":"f8c9b4d02496959e8fb124becb7d2c60f0137dcae9fa68835aac05f51b1df52b"} Oct 11 10:42:08.454721 master-1 kubenswrapper[4771]: I1011 10:42:08.454714 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" event={"ID":"6c9318b1-5be5-4254-8eb6-7cf411c02eb8","Type":"ContainerStarted","Data":"ac98991a19fe40f0e85bff03a611c5b88f0d91835e7f5d79eaa631518389ccd6"} Oct 11 10:42:09.308174 master-1 kubenswrapper[4771]: I1011 10:42:09.308063 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-etcd/installer-9-master-1"] Oct 11 10:42:09.308484 master-1 kubenswrapper[4771]: I1011 10:42:09.308328 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/installer-9-master-1" podUID="1da01b89-3c1e-4f11-bcc6-65a56654021f" containerName="installer" containerID="cri-o://1a1e8546ece3b9b09f96eb38ce98e4e2f7676e9d011955a8c3b8f572088b6cdb" gracePeriod=30 Oct 11 10:42:09.451000 master-1 kubenswrapper[4771]: I1011 10:42:09.450939 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" event={"ID":"6c9318b1-5be5-4254-8eb6-7cf411c02eb8","Type":"ContainerStarted","Data":"aa5cf014d215a1dcd8c5faa8044bde8a201a3720fd928755ca335ca36d17fdf2"} Oct 11 10:42:09.451000 master-1 kubenswrapper[4771]: I1011 10:42:09.450995 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" event={"ID":"6c9318b1-5be5-4254-8eb6-7cf411c02eb8","Type":"ContainerStarted","Data":"3d85f294797215e27e00a1deb2e17dd5dd10acdf5ffc31590c560bc071b3ebc4"} Oct 11 10:42:09.485751 master-1 kubenswrapper[4771]: I1011 10:42:09.485663 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" podStartSLOduration=21.485644522 podStartE2EDuration="21.485644522s" podCreationTimestamp="2025-10-11 10:41:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:42:09.47831047 +0000 UTC m=+961.452536911" watchObservedRunningTime="2025-10-11 10:42:09.485644522 +0000 UTC m=+961.459870963" Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: I1011 10:42:10.906069 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:42:10.906200 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:42:10.907509 master-1 kubenswrapper[4771]: I1011 10:42:10.906911 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:42:12.467546 master-1 kubenswrapper[4771]: I1011 10:42:12.467481 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-etcd/installer-10-master-1"] Oct 11 10:42:12.471075 master-1 kubenswrapper[4771]: I1011 10:42:12.471005 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.486460 master-1 kubenswrapper[4771]: I1011 10:42:12.485915 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/installer-10-master-1"] Oct 11 10:42:12.550989 master-1 kubenswrapper[4771]: I1011 10:42:12.550886 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:12.550989 master-1 kubenswrapper[4771]: I1011 10:42:12.550984 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:12.561062 master-1 kubenswrapper[4771]: I1011 10:42:12.560981 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:12.603723 master-1 kubenswrapper[4771]: I1011 10:42:12.603671 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-kubelet-dir\") pod \"installer-10-master-1\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.604078 master-1 kubenswrapper[4771]: I1011 10:42:12.604050 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/8d7775e5-5c08-4eef-84bf-8995a11eb190-kube-api-access\") pod \"installer-10-master-1\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.604256 master-1 kubenswrapper[4771]: I1011 10:42:12.604235 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-var-lock\") pod \"installer-10-master-1\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.706453 master-1 kubenswrapper[4771]: I1011 10:42:12.706331 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-kubelet-dir\") pod \"installer-10-master-1\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.706718 master-1 kubenswrapper[4771]: I1011 10:42:12.706485 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-kubelet-dir\") pod \"installer-10-master-1\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.706718 master-1 kubenswrapper[4771]: I1011 10:42:12.706670 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/8d7775e5-5c08-4eef-84bf-8995a11eb190-kube-api-access\") pod \"installer-10-master-1\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.706941 master-1 kubenswrapper[4771]: I1011 10:42:12.706886 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-var-lock\") pod \"installer-10-master-1\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.707076 master-1 kubenswrapper[4771]: I1011 10:42:12.707024 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-var-lock\") pod \"installer-10-master-1\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.740090 master-1 kubenswrapper[4771]: I1011 10:42:12.739964 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/8d7775e5-5c08-4eef-84bf-8995a11eb190-kube-api-access\") pod \"installer-10-master-1\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:12.785832 master-1 kubenswrapper[4771]: I1011 10:42:12.785754 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-10-master-1" Oct 11 10:42:13.322680 master-1 kubenswrapper[4771]: I1011 10:42:13.322580 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/installer-10-master-1"] Oct 11 10:42:13.483567 master-1 kubenswrapper[4771]: I1011 10:42:13.483492 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-10-master-1" event={"ID":"8d7775e5-5c08-4eef-84bf-8995a11eb190","Type":"ContainerStarted","Data":"16353a00dd4456281d0e795316b26f0bbce37de72b33a0538c00a1e5b2391471"} Oct 11 10:42:13.490194 master-1 kubenswrapper[4771]: I1011 10:42:13.490155 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-apiserver/apiserver-8865994fd-g2fnh" Oct 11 10:42:14.490164 master-1 kubenswrapper[4771]: I1011 10:42:14.490052 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-10-master-1" event={"ID":"8d7775e5-5c08-4eef-84bf-8995a11eb190","Type":"ContainerStarted","Data":"ea6046ea85f7a0fce021fb5f4d0cfe1454a1393bcf7a0d41b1a58c6b303f5dca"} Oct 11 10:42:14.520912 master-1 kubenswrapper[4771]: I1011 10:42:14.520810 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-etcd/installer-10-master-1" podStartSLOduration=2.520788535 podStartE2EDuration="2.520788535s" podCreationTimestamp="2025-10-11 10:42:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:42:14.515893473 +0000 UTC m=+966.490119934" watchObservedRunningTime="2025-10-11 10:42:14.520788535 +0000 UTC m=+966.495014996" Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: I1011 10:42:15.908892 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:42:15.909001 master-1 kubenswrapper[4771]: I1011 10:42:15.908984 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: I1011 10:42:20.908614 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:42:20.908746 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:42:20.910859 master-1 kubenswrapper[4771]: I1011 10:42:20.908746 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: I1011 10:42:25.906775 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:42:25.906869 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:42:25.908519 master-1 kubenswrapper[4771]: I1011 10:42:25.906874 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: I1011 10:42:30.907974 4771 patch_prober.go:28] interesting pod/apiserver-656768b4df-g4p26 container/oauth-apiserver namespace/openshift-oauth-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]etcd excluded: ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]etcd-readiness excluded: ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]poststarthook/max-in-flight-filter ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartUserInformer ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartOAuthInformer ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-StartTokenTimeoutUpdater ok Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:42:30.908060 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:42:30.910171 master-1 kubenswrapper[4771]: I1011 10:42:30.908150 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:42:33.639282 master-1 kubenswrapper[4771]: I1011 10:42:33.639205 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_installer-9-master-1_1da01b89-3c1e-4f11-bcc6-65a56654021f/installer/0.log" Oct 11 10:42:33.640050 master-1 kubenswrapper[4771]: I1011 10:42:33.639438 4771 generic.go:334] "Generic (PLEG): container finished" podID="1da01b89-3c1e-4f11-bcc6-65a56654021f" containerID="1a1e8546ece3b9b09f96eb38ce98e4e2f7676e9d011955a8c3b8f572088b6cdb" exitCode=1 Oct 11 10:42:33.640050 master-1 kubenswrapper[4771]: I1011 10:42:33.639520 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-9-master-1" event={"ID":"1da01b89-3c1e-4f11-bcc6-65a56654021f","Type":"ContainerDied","Data":"1a1e8546ece3b9b09f96eb38ce98e4e2f7676e9d011955a8c3b8f572088b6cdb"} Oct 11 10:42:34.048890 master-1 kubenswrapper[4771]: I1011 10:42:34.047900 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_installer-9-master-1_1da01b89-3c1e-4f11-bcc6-65a56654021f/installer/0.log" Oct 11 10:42:34.048890 master-1 kubenswrapper[4771]: I1011 10:42:34.048081 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:34.075542 master-1 kubenswrapper[4771]: I1011 10:42:34.075482 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/1da01b89-3c1e-4f11-bcc6-65a56654021f-kube-api-access\") pod \"1da01b89-3c1e-4f11-bcc6-65a56654021f\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " Oct 11 10:42:34.075854 master-1 kubenswrapper[4771]: I1011 10:42:34.075821 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-kubelet-dir\") pod \"1da01b89-3c1e-4f11-bcc6-65a56654021f\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " Oct 11 10:42:34.075942 master-1 kubenswrapper[4771]: I1011 10:42:34.075872 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-var-lock\") pod \"1da01b89-3c1e-4f11-bcc6-65a56654021f\" (UID: \"1da01b89-3c1e-4f11-bcc6-65a56654021f\") " Oct 11 10:42:34.076129 master-1 kubenswrapper[4771]: I1011 10:42:34.076066 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "1da01b89-3c1e-4f11-bcc6-65a56654021f" (UID: "1da01b89-3c1e-4f11-bcc6-65a56654021f"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:42:34.076380 master-1 kubenswrapper[4771]: I1011 10:42:34.076249 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-var-lock" (OuterVolumeSpecName: "var-lock") pod "1da01b89-3c1e-4f11-bcc6-65a56654021f" (UID: "1da01b89-3c1e-4f11-bcc6-65a56654021f"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:42:34.077797 master-1 kubenswrapper[4771]: I1011 10:42:34.077150 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.077797 master-1 kubenswrapper[4771]: I1011 10:42:34.077255 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1da01b89-3c1e-4f11-bcc6-65a56654021f-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.088327 master-1 kubenswrapper[4771]: I1011 10:42:34.088221 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da01b89-3c1e-4f11-bcc6-65a56654021f-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "1da01b89-3c1e-4f11-bcc6-65a56654021f" (UID: "1da01b89-3c1e-4f11-bcc6-65a56654021f"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:42:34.158001 master-1 kubenswrapper[4771]: I1011 10:42:34.157348 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:42:34.178948 master-1 kubenswrapper[4771]: I1011 10:42:34.178872 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/1da01b89-3c1e-4f11-bcc6-65a56654021f-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.280065 master-1 kubenswrapper[4771]: I1011 10:42:34.279970 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-encryption-config\") pod \"4c0cf305-ba21-45c0-a092-05214809da68\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " Oct 11 10:42:34.280415 master-1 kubenswrapper[4771]: I1011 10:42:34.280086 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-etcd-serving-ca\") pod \"4c0cf305-ba21-45c0-a092-05214809da68\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " Oct 11 10:42:34.280415 master-1 kubenswrapper[4771]: I1011 10:42:34.280151 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjn45\" (UniqueName: \"kubernetes.io/projected/4c0cf305-ba21-45c0-a092-05214809da68-kube-api-access-rjn45\") pod \"4c0cf305-ba21-45c0-a092-05214809da68\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " Oct 11 10:42:34.280415 master-1 kubenswrapper[4771]: I1011 10:42:34.280201 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/4c0cf305-ba21-45c0-a092-05214809da68-audit-dir\") pod \"4c0cf305-ba21-45c0-a092-05214809da68\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " Oct 11 10:42:34.280415 master-1 kubenswrapper[4771]: I1011 10:42:34.280239 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-etcd-client\") pod \"4c0cf305-ba21-45c0-a092-05214809da68\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " Oct 11 10:42:34.280415 master-1 kubenswrapper[4771]: I1011 10:42:34.280278 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-trusted-ca-bundle\") pod \"4c0cf305-ba21-45c0-a092-05214809da68\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " Oct 11 10:42:34.280415 master-1 kubenswrapper[4771]: I1011 10:42:34.280319 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-serving-cert\") pod \"4c0cf305-ba21-45c0-a092-05214809da68\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " Oct 11 10:42:34.280415 master-1 kubenswrapper[4771]: I1011 10:42:34.280379 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-audit-policies\") pod \"4c0cf305-ba21-45c0-a092-05214809da68\" (UID: \"4c0cf305-ba21-45c0-a092-05214809da68\") " Oct 11 10:42:34.281773 master-1 kubenswrapper[4771]: I1011 10:42:34.280534 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/4c0cf305-ba21-45c0-a092-05214809da68-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "4c0cf305-ba21-45c0-a092-05214809da68" (UID: "4c0cf305-ba21-45c0-a092-05214809da68"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:42:34.282320 master-1 kubenswrapper[4771]: I1011 10:42:34.282241 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-audit-policies" (OuterVolumeSpecName: "audit-policies") pod "4c0cf305-ba21-45c0-a092-05214809da68" (UID: "4c0cf305-ba21-45c0-a092-05214809da68"). InnerVolumeSpecName "audit-policies". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:42:34.282320 master-1 kubenswrapper[4771]: I1011 10:42:34.282268 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-etcd-serving-ca" (OuterVolumeSpecName: "etcd-serving-ca") pod "4c0cf305-ba21-45c0-a092-05214809da68" (UID: "4c0cf305-ba21-45c0-a092-05214809da68"). InnerVolumeSpecName "etcd-serving-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:42:34.283732 master-1 kubenswrapper[4771]: I1011 10:42:34.283668 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-etcd-serving-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.283732 master-1 kubenswrapper[4771]: I1011 10:42:34.283721 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/4c0cf305-ba21-45c0-a092-05214809da68-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.283953 master-1 kubenswrapper[4771]: I1011 10:42:34.283749 4771 reconciler_common.go:293] "Volume detached for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-audit-policies\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.285708 master-1 kubenswrapper[4771]: I1011 10:42:34.285613 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-etcd-client" (OuterVolumeSpecName: "etcd-client") pod "4c0cf305-ba21-45c0-a092-05214809da68" (UID: "4c0cf305-ba21-45c0-a092-05214809da68"). InnerVolumeSpecName "etcd-client". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:42:34.286174 master-1 kubenswrapper[4771]: I1011 10:42:34.286109 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-encryption-config" (OuterVolumeSpecName: "encryption-config") pod "4c0cf305-ba21-45c0-a092-05214809da68" (UID: "4c0cf305-ba21-45c0-a092-05214809da68"). InnerVolumeSpecName "encryption-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:42:34.286819 master-1 kubenswrapper[4771]: I1011 10:42:34.286757 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-serving-cert" (OuterVolumeSpecName: "serving-cert") pod "4c0cf305-ba21-45c0-a092-05214809da68" (UID: "4c0cf305-ba21-45c0-a092-05214809da68"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:42:34.288166 master-1 kubenswrapper[4771]: I1011 10:42:34.288093 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4c0cf305-ba21-45c0-a092-05214809da68" (UID: "4c0cf305-ba21-45c0-a092-05214809da68"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:42:34.293507 master-1 kubenswrapper[4771]: I1011 10:42:34.290578 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0cf305-ba21-45c0-a092-05214809da68-kube-api-access-rjn45" (OuterVolumeSpecName: "kube-api-access-rjn45") pod "4c0cf305-ba21-45c0-a092-05214809da68" (UID: "4c0cf305-ba21-45c0-a092-05214809da68"). InnerVolumeSpecName "kube-api-access-rjn45". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:42:34.385512 master-1 kubenswrapper[4771]: I1011 10:42:34.385419 4771 reconciler_common.go:293] "Volume detached for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-encryption-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.385512 master-1 kubenswrapper[4771]: I1011 10:42:34.385488 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-rjn45\" (UniqueName: \"kubernetes.io/projected/4c0cf305-ba21-45c0-a092-05214809da68-kube-api-access-rjn45\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.385512 master-1 kubenswrapper[4771]: I1011 10:42:34.385510 4771 reconciler_common.go:293] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-etcd-client\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.385512 master-1 kubenswrapper[4771]: I1011 10:42:34.385530 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c0cf305-ba21-45c0-a092-05214809da68-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.385998 master-1 kubenswrapper[4771]: I1011 10:42:34.385549 4771 reconciler_common.go:293] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c0cf305-ba21-45c0-a092-05214809da68-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:42:34.654443 master-1 kubenswrapper[4771]: I1011 10:42:34.654340 4771 generic.go:334] "Generic (PLEG): container finished" podID="4c0cf305-ba21-45c0-a092-05214809da68" containerID="0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948" exitCode=0 Oct 11 10:42:34.655222 master-1 kubenswrapper[4771]: I1011 10:42:34.654427 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" event={"ID":"4c0cf305-ba21-45c0-a092-05214809da68","Type":"ContainerDied","Data":"0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948"} Oct 11 10:42:34.655222 master-1 kubenswrapper[4771]: I1011 10:42:34.654532 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" Oct 11 10:42:34.655222 master-1 kubenswrapper[4771]: I1011 10:42:34.654559 4771 scope.go:117] "RemoveContainer" containerID="0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948" Oct 11 10:42:34.655222 master-1 kubenswrapper[4771]: I1011 10:42:34.654538 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-656768b4df-g4p26" event={"ID":"4c0cf305-ba21-45c0-a092-05214809da68","Type":"ContainerDied","Data":"9a8773a82720172e1c708c6b8b379786c06f2193ace376125888c909cd115b04"} Oct 11 10:42:34.661163 master-1 kubenswrapper[4771]: I1011 10:42:34.661100 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_installer-9-master-1_1da01b89-3c1e-4f11-bcc6-65a56654021f/installer/0.log" Oct 11 10:42:34.661299 master-1 kubenswrapper[4771]: I1011 10:42:34.661189 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-9-master-1" event={"ID":"1da01b89-3c1e-4f11-bcc6-65a56654021f","Type":"ContainerDied","Data":"9692635e40b2a711a263503ff5795f641f5480d42c4c64a94f91d9bd4aff98f6"} Oct 11 10:42:34.661724 master-1 kubenswrapper[4771]: I1011 10:42:34.661593 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-9-master-1" Oct 11 10:42:34.681728 master-1 kubenswrapper[4771]: I1011 10:42:34.681658 4771 scope.go:117] "RemoveContainer" containerID="0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22" Oct 11 10:42:34.695119 master-1 kubenswrapper[4771]: I1011 10:42:34.695066 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-oauth-apiserver/apiserver-656768b4df-g4p26"] Oct 11 10:42:34.711075 master-1 kubenswrapper[4771]: I1011 10:42:34.711005 4771 scope.go:117] "RemoveContainer" containerID="0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948" Oct 11 10:42:34.711851 master-1 kubenswrapper[4771]: E1011 10:42:34.711655 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948\": container with ID starting with 0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948 not found: ID does not exist" containerID="0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948" Oct 11 10:42:34.711851 master-1 kubenswrapper[4771]: I1011 10:42:34.711710 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948"} err="failed to get container status \"0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948\": rpc error: code = NotFound desc = could not find container \"0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948\": container with ID starting with 0d35eb346e00db345ba38192641eb8af77030ecba70c39b9071f558249b90948 not found: ID does not exist" Oct 11 10:42:34.711851 master-1 kubenswrapper[4771]: I1011 10:42:34.711742 4771 scope.go:117] "RemoveContainer" containerID="0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22" Oct 11 10:42:34.712899 master-1 kubenswrapper[4771]: E1011 10:42:34.712536 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22\": container with ID starting with 0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22 not found: ID does not exist" containerID="0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22" Oct 11 10:42:34.712899 master-1 kubenswrapper[4771]: I1011 10:42:34.712617 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22"} err="failed to get container status \"0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22\": rpc error: code = NotFound desc = could not find container \"0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22\": container with ID starting with 0e1c25b025232015baea7f43e82a2fb07a8814a3aa56b9f5440885f746e96d22 not found: ID does not exist" Oct 11 10:42:34.712899 master-1 kubenswrapper[4771]: I1011 10:42:34.712671 4771 scope.go:117] "RemoveContainer" containerID="1a1e8546ece3b9b09f96eb38ce98e4e2f7676e9d011955a8c3b8f572088b6cdb" Oct 11 10:42:34.719248 master-1 kubenswrapper[4771]: I1011 10:42:34.719167 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-oauth-apiserver/apiserver-656768b4df-g4p26"] Oct 11 10:42:34.732407 master-1 kubenswrapper[4771]: I1011 10:42:34.732329 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-etcd/installer-9-master-1"] Oct 11 10:42:34.738179 master-1 kubenswrapper[4771]: I1011 10:42:34.737169 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-etcd/installer-9-master-1"] Oct 11 10:42:36.449812 master-1 kubenswrapper[4771]: I1011 10:42:36.449698 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da01b89-3c1e-4f11-bcc6-65a56654021f" path="/var/lib/kubelet/pods/1da01b89-3c1e-4f11-bcc6-65a56654021f/volumes" Oct 11 10:42:36.451042 master-1 kubenswrapper[4771]: I1011 10:42:36.450978 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c0cf305-ba21-45c0-a092-05214809da68" path="/var/lib/kubelet/pods/4c0cf305-ba21-45c0-a092-05214809da68/volumes" Oct 11 10:42:44.639506 master-1 kubenswrapper[4771]: I1011 10:42:44.639436 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll"] Oct 11 10:42:44.640259 master-1 kubenswrapper[4771]: E1011 10:42:44.639769 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="1da01b89-3c1e-4f11-bcc6-65a56654021f" containerName="installer" Oct 11 10:42:44.640259 master-1 kubenswrapper[4771]: I1011 10:42:44.639792 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da01b89-3c1e-4f11-bcc6-65a56654021f" containerName="installer" Oct 11 10:42:44.640259 master-1 kubenswrapper[4771]: E1011 10:42:44.639817 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="fix-audit-permissions" Oct 11 10:42:44.640259 master-1 kubenswrapper[4771]: I1011 10:42:44.639831 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="fix-audit-permissions" Oct 11 10:42:44.640259 master-1 kubenswrapper[4771]: E1011 10:42:44.639860 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" Oct 11 10:42:44.640259 master-1 kubenswrapper[4771]: I1011 10:42:44.639874 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" Oct 11 10:42:44.640259 master-1 kubenswrapper[4771]: I1011 10:42:44.640066 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="1da01b89-3c1e-4f11-bcc6-65a56654021f" containerName="installer" Oct 11 10:42:44.640259 master-1 kubenswrapper[4771]: I1011 10:42:44.640092 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="4c0cf305-ba21-45c0-a092-05214809da68" containerName="oauth-apiserver" Oct 11 10:42:44.641309 master-1 kubenswrapper[4771]: I1011 10:42:44.641273 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.644982 master-1 kubenswrapper[4771]: I1011 10:42:44.644927 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"etcd-serving-ca" Oct 11 10:42:44.645345 master-1 kubenswrapper[4771]: I1011 10:42:44.645163 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"oauth-apiserver-sa-dockercfg-zlnjr" Oct 11 10:42:44.646161 master-1 kubenswrapper[4771]: I1011 10:42:44.646123 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"trusted-ca-bundle" Oct 11 10:42:44.646269 master-1 kubenswrapper[4771]: I1011 10:42:44.646229 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"kube-root-ca.crt" Oct 11 10:42:44.646907 master-1 kubenswrapper[4771]: I1011 10:42:44.646472 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"serving-cert" Oct 11 10:42:44.646907 master-1 kubenswrapper[4771]: I1011 10:42:44.646558 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"etcd-client" Oct 11 10:42:44.647179 master-1 kubenswrapper[4771]: I1011 10:42:44.647000 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"audit-1" Oct 11 10:42:44.647233 master-1 kubenswrapper[4771]: I1011 10:42:44.647197 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-oauth-apiserver"/"encryption-config-1" Oct 11 10:42:44.647449 master-1 kubenswrapper[4771]: I1011 10:42:44.646848 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-oauth-apiserver"/"openshift-service-ca.crt" Oct 11 10:42:44.700045 master-1 kubenswrapper[4771]: I1011 10:42:44.662222 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll"] Oct 11 10:42:44.747043 master-1 kubenswrapper[4771]: I1011 10:42:44.746988 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szs58\" (UniqueName: \"kubernetes.io/projected/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-kube-api-access-szs58\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.747043 master-1 kubenswrapper[4771]: I1011 10:42:44.747050 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-etcd-client\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.747318 master-1 kubenswrapper[4771]: I1011 10:42:44.747094 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-audit-policies\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.747318 master-1 kubenswrapper[4771]: I1011 10:42:44.747115 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-encryption-config\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.747318 master-1 kubenswrapper[4771]: I1011 10:42:44.747147 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-audit-dir\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.747318 master-1 kubenswrapper[4771]: I1011 10:42:44.747267 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-serving-cert\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.747521 master-1 kubenswrapper[4771]: I1011 10:42:44.747322 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-etcd-serving-ca\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.747634 master-1 kubenswrapper[4771]: I1011 10:42:44.747574 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-trusted-ca-bundle\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.849915 master-1 kubenswrapper[4771]: I1011 10:42:44.849829 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-etcd-client\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.850153 master-1 kubenswrapper[4771]: I1011 10:42:44.849924 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-audit-policies\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.850153 master-1 kubenswrapper[4771]: I1011 10:42:44.849969 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-encryption-config\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.850153 master-1 kubenswrapper[4771]: I1011 10:42:44.850040 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-audit-dir\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.850153 master-1 kubenswrapper[4771]: I1011 10:42:44.850091 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-serving-cert\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.850330 master-1 kubenswrapper[4771]: I1011 10:42:44.850147 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-etcd-serving-ca\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.850330 master-1 kubenswrapper[4771]: I1011 10:42:44.850203 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-trusted-ca-bundle\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.850330 master-1 kubenswrapper[4771]: I1011 10:42:44.850216 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-audit-dir\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.850330 master-1 kubenswrapper[4771]: I1011 10:42:44.850274 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-szs58\" (UniqueName: \"kubernetes.io/projected/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-kube-api-access-szs58\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.851521 master-1 kubenswrapper[4771]: I1011 10:42:44.851473 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-etcd-serving-ca\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.851628 master-1 kubenswrapper[4771]: I1011 10:42:44.851578 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-trusted-ca-bundle\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.851816 master-1 kubenswrapper[4771]: I1011 10:42:44.851758 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-policies\" (UniqueName: \"kubernetes.io/configmap/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-audit-policies\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.853791 master-1 kubenswrapper[4771]: I1011 10:42:44.853745 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-etcd-client\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.854747 master-1 kubenswrapper[4771]: I1011 10:42:44.854692 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-serving-cert\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.855094 master-1 kubenswrapper[4771]: I1011 10:42:44.855047 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"encryption-config\" (UniqueName: \"kubernetes.io/secret/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-encryption-config\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:44.868253 master-1 kubenswrapper[4771]: I1011 10:42:44.868107 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-szs58\" (UniqueName: \"kubernetes.io/projected/1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d-kube-api-access-szs58\") pod \"apiserver-68f4c55ff4-mmqll\" (UID: \"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d\") " pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:45.006912 master-1 kubenswrapper[4771]: I1011 10:42:45.006775 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:45.508250 master-1 kubenswrapper[4771]: I1011 10:42:45.508176 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll"] Oct 11 10:42:45.518822 master-1 kubenswrapper[4771]: W1011 10:42:45.514848 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8be6e8_cddf_46d6_a1e4_f0bdc0ae7f6d.slice/crio-97084740e6a91e89121a51817a01bece0e852c2fe34a9c54110a075f2cc1678b WatchSource:0}: Error finding container 97084740e6a91e89121a51817a01bece0e852c2fe34a9c54110a075f2cc1678b: Status 404 returned error can't find the container with id 97084740e6a91e89121a51817a01bece0e852c2fe34a9c54110a075f2cc1678b Oct 11 10:42:45.752955 master-1 kubenswrapper[4771]: I1011 10:42:45.751375 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" event={"ID":"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d","Type":"ContainerStarted","Data":"97084740e6a91e89121a51817a01bece0e852c2fe34a9c54110a075f2cc1678b"} Oct 11 10:42:46.762442 master-1 kubenswrapper[4771]: I1011 10:42:46.762335 4771 generic.go:334] "Generic (PLEG): container finished" podID="1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d" containerID="20de4b85fbe0e93509b158b1b1095dc9c627cebb3415ba32b94fcbca5b12e499" exitCode=0 Oct 11 10:42:46.762442 master-1 kubenswrapper[4771]: I1011 10:42:46.762435 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" event={"ID":"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d","Type":"ContainerDied","Data":"20de4b85fbe0e93509b158b1b1095dc9c627cebb3415ba32b94fcbca5b12e499"} Oct 11 10:42:47.771811 master-1 kubenswrapper[4771]: I1011 10:42:47.771751 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" event={"ID":"1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d","Type":"ContainerStarted","Data":"418a6b1668322b851f02d6a982854c54b624c024df481ef6044849a381c3452a"} Oct 11 10:42:47.802946 master-1 kubenswrapper[4771]: I1011 10:42:47.802879 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" podStartSLOduration=65.802860431 podStartE2EDuration="1m5.802860431s" podCreationTimestamp="2025-10-11 10:41:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:42:47.800333029 +0000 UTC m=+999.774559460" watchObservedRunningTime="2025-10-11 10:42:47.802860431 +0000 UTC m=+999.777086872" Oct 11 10:42:50.007716 master-1 kubenswrapper[4771]: I1011 10:42:50.007571 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:50.007716 master-1 kubenswrapper[4771]: I1011 10:42:50.007721 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:50.017769 master-1 kubenswrapper[4771]: I1011 10:42:50.017693 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:42:50.807872 master-1 kubenswrapper[4771]: I1011 10:42:50.807792 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-oauth-apiserver/apiserver-68f4c55ff4-mmqll" Oct 11 10:43:05.025663 master-1 kubenswrapper[4771]: I1011 10:43:05.025563 4771 kubelet.go:2431] "SyncLoop REMOVE" source="file" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: I1011 10:43:05.025700 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: E1011 10:43:05.026048 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-ensure-env-vars" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: I1011 10:43:05.026071 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-ensure-env-vars" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: E1011 10:43:05.026090 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-metrics" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: I1011 10:43:05.026103 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-metrics" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: E1011 10:43:05.026120 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-resources-copy" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: I1011 10:43:05.026135 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-resources-copy" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: E1011 10:43:05.026154 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: I1011 10:43:05.026166 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: E1011 10:43:05.026181 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-readyz" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: I1011 10:43:05.026145 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-rev" containerID="cri-o://2f39d1ed6551318e8799ea55ecdfbfe51ea2b9b7b26411631664f953b1d0e296" gracePeriod=30 Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: I1011 10:43:05.026253 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd" containerID="cri-o://1b08bbe8a016cc9703a454b83b5ccaac8367e55a0f3e2612f07c89255c5b066b" gracePeriod=30 Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: I1011 10:43:05.026197 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-readyz" Oct 11 10:43:05.026393 master-1 kubenswrapper[4771]: I1011 10:43:05.026382 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcdctl" containerID="cri-o://49bf7adabb62db980d637017833ab23f35546844d31309e50b509a3be2303a67" gracePeriod=30 Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026324 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-readyz" containerID="cri-o://84bbf7ab3fb66f6d01d7500d037317a4cb49a3eae4199b8937858e7e953c7fd3" gracePeriod=30 Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: E1011 10:43:05.026449 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="setup" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026479 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="setup" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: E1011 10:43:05.026514 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcdctl" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026522 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcdctl" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: E1011 10:43:05.026540 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-rev" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026547 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-rev" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026279 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-etcd/etcd-master-1" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-metrics" containerID="cri-o://ecbb0613c992785c9403e057fc0c874ad563e770ca35f25a2b4b2f7341f1c10c" gracePeriod=30 Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026871 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-readyz" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026889 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcdctl" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026898 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-rev" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026907 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd" Oct 11 10:43:05.026929 master-1 kubenswrapper[4771]: I1011 10:43:05.026917 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd-metrics" Oct 11 10:43:05.063928 master-1 kubenswrapper[4771]: I1011 10:43:05.063889 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-data-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.064109 master-1 kubenswrapper[4771]: I1011 10:43:05.064091 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-log-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.064231 master-1 kubenswrapper[4771]: I1011 10:43:05.064214 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-resource-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.064347 master-1 kubenswrapper[4771]: I1011 10:43:05.064329 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-static-pod-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.064503 master-1 kubenswrapper[4771]: I1011 10:43:05.064485 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-usr-local-bin\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.064617 master-1 kubenswrapper[4771]: I1011 10:43:05.064600 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-cert-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.166636 master-1 kubenswrapper[4771]: I1011 10:43:05.166524 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-log-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.166636 master-1 kubenswrapper[4771]: I1011 10:43:05.166623 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-resource-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.166897 master-1 kubenswrapper[4771]: I1011 10:43:05.166655 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-static-pod-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.166897 master-1 kubenswrapper[4771]: I1011 10:43:05.166691 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-usr-local-bin\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.166897 master-1 kubenswrapper[4771]: I1011 10:43:05.166714 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-cert-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.166897 master-1 kubenswrapper[4771]: I1011 10:43:05.166779 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-data-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.166897 master-1 kubenswrapper[4771]: I1011 10:43:05.166882 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-data-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.167197 master-1 kubenswrapper[4771]: I1011 10:43:05.166930 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-log-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.167197 master-1 kubenswrapper[4771]: I1011 10:43:05.166962 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-resource-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.167197 master-1 kubenswrapper[4771]: I1011 10:43:05.166984 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-static-pod-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.167197 master-1 kubenswrapper[4771]: I1011 10:43:05.167007 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-usr-local-bin\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.167197 master-1 kubenswrapper[4771]: I1011 10:43:05.167027 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/dbeb1098f6b7e52b91afcf2e9b50b014-cert-dir\") pod \"etcd-master-1\" (UID: \"dbeb1098f6b7e52b91afcf2e9b50b014\") " pod="openshift-etcd/etcd-master-1" Oct 11 10:43:05.507165 master-1 kubenswrapper[4771]: I1011 10:43:05.507039 4771 patch_prober.go:28] interesting pod/etcd-master-1 container/etcd namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:05.507165 master-1 kubenswrapper[4771]: I1011 10:43:05.507133 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-master-1" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" containerName="etcd" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:05.955332 master-1 kubenswrapper[4771]: I1011 10:43:05.955256 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd-rev/0.log" Oct 11 10:43:05.956999 master-1 kubenswrapper[4771]: I1011 10:43:05.956947 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd-metrics/0.log" Oct 11 10:43:05.960097 master-1 kubenswrapper[4771]: I1011 10:43:05.960042 4771 generic.go:334] "Generic (PLEG): container finished" podID="2b1859aa05c2c75eb43d086c9ccd9c86" containerID="2f39d1ed6551318e8799ea55ecdfbfe51ea2b9b7b26411631664f953b1d0e296" exitCode=2 Oct 11 10:43:05.960097 master-1 kubenswrapper[4771]: I1011 10:43:05.960091 4771 generic.go:334] "Generic (PLEG): container finished" podID="2b1859aa05c2c75eb43d086c9ccd9c86" containerID="84bbf7ab3fb66f6d01d7500d037317a4cb49a3eae4199b8937858e7e953c7fd3" exitCode=0 Oct 11 10:43:05.960286 master-1 kubenswrapper[4771]: I1011 10:43:05.960111 4771 generic.go:334] "Generic (PLEG): container finished" podID="2b1859aa05c2c75eb43d086c9ccd9c86" containerID="ecbb0613c992785c9403e057fc0c874ad563e770ca35f25a2b4b2f7341f1c10c" exitCode=2 Oct 11 10:43:05.962198 master-1 kubenswrapper[4771]: I1011 10:43:05.962139 4771 generic.go:334] "Generic (PLEG): container finished" podID="8d7775e5-5c08-4eef-84bf-8995a11eb190" containerID="ea6046ea85f7a0fce021fb5f4d0cfe1454a1393bcf7a0d41b1a58c6b303f5dca" exitCode=0 Oct 11 10:43:05.962320 master-1 kubenswrapper[4771]: I1011 10:43:05.962200 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-10-master-1" event={"ID":"8d7775e5-5c08-4eef-84bf-8995a11eb190","Type":"ContainerDied","Data":"ea6046ea85f7a0fce021fb5f4d0cfe1454a1393bcf7a0d41b1a58c6b303f5dca"} Oct 11 10:43:05.973479 master-1 kubenswrapper[4771]: I1011 10:43:05.973341 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-master-1" oldPodUID="2b1859aa05c2c75eb43d086c9ccd9c86" podUID="dbeb1098f6b7e52b91afcf2e9b50b014" Oct 11 10:43:07.444728 master-1 kubenswrapper[4771]: I1011 10:43:07.444663 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-10-master-1" Oct 11 10:43:07.600190 master-1 kubenswrapper[4771]: I1011 10:43:07.599908 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-var-lock\") pod \"8d7775e5-5c08-4eef-84bf-8995a11eb190\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " Oct 11 10:43:07.600190 master-1 kubenswrapper[4771]: I1011 10:43:07.599995 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-kubelet-dir\") pod \"8d7775e5-5c08-4eef-84bf-8995a11eb190\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " Oct 11 10:43:07.600190 master-1 kubenswrapper[4771]: I1011 10:43:07.600062 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-var-lock" (OuterVolumeSpecName: "var-lock") pod "8d7775e5-5c08-4eef-84bf-8995a11eb190" (UID: "8d7775e5-5c08-4eef-84bf-8995a11eb190"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:07.600190 master-1 kubenswrapper[4771]: I1011 10:43:07.600135 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "8d7775e5-5c08-4eef-84bf-8995a11eb190" (UID: "8d7775e5-5c08-4eef-84bf-8995a11eb190"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:07.600660 master-1 kubenswrapper[4771]: I1011 10:43:07.600216 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/8d7775e5-5c08-4eef-84bf-8995a11eb190-kube-api-access\") pod \"8d7775e5-5c08-4eef-84bf-8995a11eb190\" (UID: \"8d7775e5-5c08-4eef-84bf-8995a11eb190\") " Oct 11 10:43:07.600660 master-1 kubenswrapper[4771]: I1011 10:43:07.600599 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:07.600660 master-1 kubenswrapper[4771]: I1011 10:43:07.600621 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d7775e5-5c08-4eef-84bf-8995a11eb190-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:07.605548 master-1 kubenswrapper[4771]: I1011 10:43:07.605497 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7775e5-5c08-4eef-84bf-8995a11eb190-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "8d7775e5-5c08-4eef-84bf-8995a11eb190" (UID: "8d7775e5-5c08-4eef-84bf-8995a11eb190"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:43:07.701724 master-1 kubenswrapper[4771]: I1011 10:43:07.701595 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/8d7775e5-5c08-4eef-84bf-8995a11eb190-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:07.980469 master-1 kubenswrapper[4771]: I1011 10:43:07.980225 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/installer-10-master-1" event={"ID":"8d7775e5-5c08-4eef-84bf-8995a11eb190","Type":"ContainerDied","Data":"16353a00dd4456281d0e795316b26f0bbce37de72b33a0538c00a1e5b2391471"} Oct 11 10:43:07.980469 master-1 kubenswrapper[4771]: I1011 10:43:07.980322 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16353a00dd4456281d0e795316b26f0bbce37de72b33a0538c00a1e5b2391471" Oct 11 10:43:07.980469 master-1 kubenswrapper[4771]: I1011 10:43:07.980386 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/installer-10-master-1" Oct 11 10:43:09.629231 master-1 kubenswrapper[4771]: I1011 10:43:09.629082 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:09.630173 master-1 kubenswrapper[4771]: I1011 10:43:09.629241 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:10.752733 master-1 kubenswrapper[4771]: I1011 10:43:10.752588 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/installer-6-master-1"] Oct 11 10:43:10.754027 master-1 kubenswrapper[4771]: E1011 10:43:10.753928 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="8d7775e5-5c08-4eef-84bf-8995a11eb190" containerName="installer" Oct 11 10:43:10.754027 master-1 kubenswrapper[4771]: I1011 10:43:10.754008 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d7775e5-5c08-4eef-84bf-8995a11eb190" containerName="installer" Oct 11 10:43:10.754249 master-1 kubenswrapper[4771]: I1011 10:43:10.754171 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="8d7775e5-5c08-4eef-84bf-8995a11eb190" containerName="installer" Oct 11 10:43:10.755090 master-1 kubenswrapper[4771]: I1011 10:43:10.755034 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:10.759896 master-1 kubenswrapper[4771]: I1011 10:43:10.759794 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-kube-controller-manager"/"installer-sa-dockercfg-djvlq" Oct 11 10:43:10.766123 master-1 kubenswrapper[4771]: I1011 10:43:10.766052 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-6-master-1"] Oct 11 10:43:10.958070 master-1 kubenswrapper[4771]: I1011 10:43:10.957946 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/78689bdc-0258-45eb-8e5b-253911c61c79-kube-api-access\") pod \"installer-6-master-1\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:10.958070 master-1 kubenswrapper[4771]: I1011 10:43:10.958065 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-kubelet-dir\") pod \"installer-6-master-1\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:10.958486 master-1 kubenswrapper[4771]: I1011 10:43:10.958102 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-var-lock\") pod \"installer-6-master-1\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:11.059668 master-1 kubenswrapper[4771]: I1011 10:43:11.059559 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/78689bdc-0258-45eb-8e5b-253911c61c79-kube-api-access\") pod \"installer-6-master-1\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:11.059914 master-1 kubenswrapper[4771]: I1011 10:43:11.059716 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-kubelet-dir\") pod \"installer-6-master-1\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:11.059914 master-1 kubenswrapper[4771]: I1011 10:43:11.059754 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-var-lock\") pod \"installer-6-master-1\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:11.060154 master-1 kubenswrapper[4771]: I1011 10:43:11.059919 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-var-lock\") pod \"installer-6-master-1\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:11.060154 master-1 kubenswrapper[4771]: I1011 10:43:11.059928 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-kubelet-dir\") pod \"installer-6-master-1\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:11.097808 master-1 kubenswrapper[4771]: I1011 10:43:11.097716 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/78689bdc-0258-45eb-8e5b-253911c61c79-kube-api-access\") pod \"installer-6-master-1\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:11.386404 master-1 kubenswrapper[4771]: I1011 10:43:11.386193 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:11.857424 master-1 kubenswrapper[4771]: I1011 10:43:11.857340 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-controller-manager/installer-6-master-1"] Oct 11 10:43:11.866929 master-1 kubenswrapper[4771]: W1011 10:43:11.866793 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod78689bdc_0258_45eb_8e5b_253911c61c79.slice/crio-56046807ca0aa308b5de3a340c0e367a86144d71f324a1b35aa3b72ae51dc090 WatchSource:0}: Error finding container 56046807ca0aa308b5de3a340c0e367a86144d71f324a1b35aa3b72ae51dc090: Status 404 returned error can't find the container with id 56046807ca0aa308b5de3a340c0e367a86144d71f324a1b35aa3b72ae51dc090 Oct 11 10:43:12.023692 master-1 kubenswrapper[4771]: I1011 10:43:12.023593 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-6-master-1" event={"ID":"78689bdc-0258-45eb-8e5b-253911c61c79","Type":"ContainerStarted","Data":"56046807ca0aa308b5de3a340c0e367a86144d71f324a1b35aa3b72ae51dc090"} Oct 11 10:43:13.034666 master-1 kubenswrapper[4771]: I1011 10:43:13.034488 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-6-master-1" event={"ID":"78689bdc-0258-45eb-8e5b-253911c61c79","Type":"ContainerStarted","Data":"2d907b9a8cd0470d88178cfea01b0abf30291128bc9c158e361b094caee83ec4"} Oct 11 10:43:13.059468 master-1 kubenswrapper[4771]: I1011 10:43:13.059292 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/installer-6-master-1" podStartSLOduration=3.059263036 podStartE2EDuration="3.059263036s" podCreationTimestamp="2025-10-11 10:43:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:43:13.05730207 +0000 UTC m=+1025.031528601" watchObservedRunningTime="2025-10-11 10:43:13.059263036 +0000 UTC m=+1025.033489517" Oct 11 10:43:14.628300 master-1 kubenswrapper[4771]: I1011 10:43:14.628229 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:14.628971 master-1 kubenswrapper[4771]: I1011 10:43:14.628328 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:19.628744 master-1 kubenswrapper[4771]: I1011 10:43:19.628595 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:19.628744 master-1 kubenswrapper[4771]: I1011 10:43:19.628730 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:19.630247 master-1 kubenswrapper[4771]: I1011 10:43:19.628866 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:43:19.630247 master-1 kubenswrapper[4771]: I1011 10:43:19.629830 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:19.630247 master-1 kubenswrapper[4771]: I1011 10:43:19.629878 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:24.628883 master-1 kubenswrapper[4771]: I1011 10:43:24.628773 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:24.629806 master-1 kubenswrapper[4771]: I1011 10:43:24.628911 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:29.629024 master-1 kubenswrapper[4771]: I1011 10:43:29.628934 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:29.629842 master-1 kubenswrapper[4771]: I1011 10:43:29.629020 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:34.628816 master-1 kubenswrapper[4771]: I1011 10:43:34.628699 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:34.628816 master-1 kubenswrapper[4771]: I1011 10:43:34.628786 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:35.203222 master-1 kubenswrapper[4771]: I1011 10:43:35.203152 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd-rev/0.log" Oct 11 10:43:35.204161 master-1 kubenswrapper[4771]: I1011 10:43:35.204124 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd-metrics/0.log" Oct 11 10:43:35.204856 master-1 kubenswrapper[4771]: I1011 10:43:35.204825 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd/0.log" Oct 11 10:43:35.205445 master-1 kubenswrapper[4771]: I1011 10:43:35.205385 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcdctl/0.log" Oct 11 10:43:35.206822 master-1 kubenswrapper[4771]: I1011 10:43:35.206770 4771 generic.go:334] "Generic (PLEG): container finished" podID="2b1859aa05c2c75eb43d086c9ccd9c86" containerID="1b08bbe8a016cc9703a454b83b5ccaac8367e55a0f3e2612f07c89255c5b066b" exitCode=137 Oct 11 10:43:35.206822 master-1 kubenswrapper[4771]: I1011 10:43:35.206802 4771 generic.go:334] "Generic (PLEG): container finished" podID="2b1859aa05c2c75eb43d086c9ccd9c86" containerID="49bf7adabb62db980d637017833ab23f35546844d31309e50b509a3be2303a67" exitCode=137 Oct 11 10:43:35.616022 master-1 kubenswrapper[4771]: I1011 10:43:35.615960 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd-rev/0.log" Oct 11 10:43:35.617259 master-1 kubenswrapper[4771]: I1011 10:43:35.617229 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd-metrics/0.log" Oct 11 10:43:35.618970 master-1 kubenswrapper[4771]: I1011 10:43:35.618936 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd/0.log" Oct 11 10:43:35.619481 master-1 kubenswrapper[4771]: I1011 10:43:35.619453 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcdctl/0.log" Oct 11 10:43:35.620764 master-1 kubenswrapper[4771]: I1011 10:43:35.620733 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:43:35.625599 master-1 kubenswrapper[4771]: I1011 10:43:35.625531 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-master-1" oldPodUID="2b1859aa05c2c75eb43d086c9ccd9c86" podUID="dbeb1098f6b7e52b91afcf2e9b50b014" Oct 11 10:43:35.681798 master-1 kubenswrapper[4771]: I1011 10:43:35.681745 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-log-dir\") pod \"2b1859aa05c2c75eb43d086c9ccd9c86\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.681808 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-data-dir\") pod \"2b1859aa05c2c75eb43d086c9ccd9c86\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.681830 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-cert-dir\") pod \"2b1859aa05c2c75eb43d086c9ccd9c86\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.681851 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-resource-dir\") pod \"2b1859aa05c2c75eb43d086c9ccd9c86\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.681887 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-static-pod-dir\") pod \"2b1859aa05c2c75eb43d086c9ccd9c86\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.681884 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-log-dir" (OuterVolumeSpecName: "log-dir") pod "2b1859aa05c2c75eb43d086c9ccd9c86" (UID: "2b1859aa05c2c75eb43d086c9ccd9c86"). InnerVolumeSpecName "log-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.681937 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-usr-local-bin" (OuterVolumeSpecName: "usr-local-bin") pod "2b1859aa05c2c75eb43d086c9ccd9c86" (UID: "2b1859aa05c2c75eb43d086c9ccd9c86"). InnerVolumeSpecName "usr-local-bin". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.681914 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-usr-local-bin\") pod \"2b1859aa05c2c75eb43d086c9ccd9c86\" (UID: \"2b1859aa05c2c75eb43d086c9ccd9c86\") " Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.681953 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-data-dir" (OuterVolumeSpecName: "data-dir") pod "2b1859aa05c2c75eb43d086c9ccd9c86" (UID: "2b1859aa05c2c75eb43d086c9ccd9c86"). InnerVolumeSpecName "data-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.682014 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-cert-dir" (OuterVolumeSpecName: "cert-dir") pod "2b1859aa05c2c75eb43d086c9ccd9c86" (UID: "2b1859aa05c2c75eb43d086c9ccd9c86"). InnerVolumeSpecName "cert-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.682056 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-static-pod-dir" (OuterVolumeSpecName: "static-pod-dir") pod "2b1859aa05c2c75eb43d086c9ccd9c86" (UID: "2b1859aa05c2c75eb43d086c9ccd9c86"). InnerVolumeSpecName "static-pod-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:35.682129 master-1 kubenswrapper[4771]: I1011 10:43:35.681974 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-resource-dir" (OuterVolumeSpecName: "resource-dir") pod "2b1859aa05c2c75eb43d086c9ccd9c86" (UID: "2b1859aa05c2c75eb43d086c9ccd9c86"). InnerVolumeSpecName "resource-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:35.682654 master-1 kubenswrapper[4771]: I1011 10:43:35.682606 4771 reconciler_common.go:293] "Volume detached for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-data-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:35.682693 master-1 kubenswrapper[4771]: I1011 10:43:35.682658 4771 reconciler_common.go:293] "Volume detached for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-cert-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:35.682693 master-1 kubenswrapper[4771]: I1011 10:43:35.682686 4771 reconciler_common.go:293] "Volume detached for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-resource-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:35.682755 master-1 kubenswrapper[4771]: I1011 10:43:35.682717 4771 reconciler_common.go:293] "Volume detached for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-static-pod-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:35.682755 master-1 kubenswrapper[4771]: I1011 10:43:35.682745 4771 reconciler_common.go:293] "Volume detached for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-usr-local-bin\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:35.682816 master-1 kubenswrapper[4771]: I1011 10:43:35.682768 4771 reconciler_common.go:293] "Volume detached for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/2b1859aa05c2c75eb43d086c9ccd9c86-log-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:36.219164 master-1 kubenswrapper[4771]: I1011 10:43:36.219065 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd-rev/0.log" Oct 11 10:43:36.221214 master-1 kubenswrapper[4771]: I1011 10:43:36.221119 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd-metrics/0.log" Oct 11 10:43:36.222498 master-1 kubenswrapper[4771]: I1011 10:43:36.222445 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcd/0.log" Oct 11 10:43:36.223311 master-1 kubenswrapper[4771]: I1011 10:43:36.223250 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_2b1859aa05c2c75eb43d086c9ccd9c86/etcdctl/0.log" Oct 11 10:43:36.225776 master-1 kubenswrapper[4771]: I1011 10:43:36.225723 4771 scope.go:117] "RemoveContainer" containerID="2f39d1ed6551318e8799ea55ecdfbfe51ea2b9b7b26411631664f953b1d0e296" Oct 11 10:43:36.225984 master-1 kubenswrapper[4771]: I1011 10:43:36.225847 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:43:36.234671 master-1 kubenswrapper[4771]: I1011 10:43:36.234479 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-master-1" oldPodUID="2b1859aa05c2c75eb43d086c9ccd9c86" podUID="dbeb1098f6b7e52b91afcf2e9b50b014" Oct 11 10:43:36.260526 master-1 kubenswrapper[4771]: I1011 10:43:36.260469 4771 scope.go:117] "RemoveContainer" containerID="84bbf7ab3fb66f6d01d7500d037317a4cb49a3eae4199b8937858e7e953c7fd3" Oct 11 10:43:36.268623 master-1 kubenswrapper[4771]: I1011 10:43:36.268559 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-master-1" oldPodUID="2b1859aa05c2c75eb43d086c9ccd9c86" podUID="dbeb1098f6b7e52b91afcf2e9b50b014" Oct 11 10:43:36.289664 master-1 kubenswrapper[4771]: I1011 10:43:36.289602 4771 scope.go:117] "RemoveContainer" containerID="ecbb0613c992785c9403e057fc0c874ad563e770ca35f25a2b4b2f7341f1c10c" Oct 11 10:43:36.310244 master-1 kubenswrapper[4771]: I1011 10:43:36.310183 4771 scope.go:117] "RemoveContainer" containerID="1b08bbe8a016cc9703a454b83b5ccaac8367e55a0f3e2612f07c89255c5b066b" Oct 11 10:43:36.334087 master-1 kubenswrapper[4771]: I1011 10:43:36.334032 4771 scope.go:117] "RemoveContainer" containerID="49bf7adabb62db980d637017833ab23f35546844d31309e50b509a3be2303a67" Oct 11 10:43:36.349420 master-1 kubenswrapper[4771]: I1011 10:43:36.349340 4771 scope.go:117] "RemoveContainer" containerID="0d2abececcc3750380edf401f993d45ec701aaab0b1cc115175ab53e903df0d6" Oct 11 10:43:36.372300 master-1 kubenswrapper[4771]: I1011 10:43:36.372251 4771 scope.go:117] "RemoveContainer" containerID="f36eed4b60a75dfc18926f5f7a62c7fe09c6ef035bfef9182c1502b7c4eeb07b" Oct 11 10:43:36.402335 master-1 kubenswrapper[4771]: I1011 10:43:36.402293 4771 scope.go:117] "RemoveContainer" containerID="5df2d69fcce5aa4d0f872e664dab924a82b358ddfdc487a9796493b554db07ec" Oct 11 10:43:36.447900 master-1 kubenswrapper[4771]: I1011 10:43:36.447831 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1859aa05c2c75eb43d086c9ccd9c86" path="/var/lib/kubelet/pods/2b1859aa05c2c75eb43d086c9ccd9c86/volumes" Oct 11 10:43:39.629485 master-1 kubenswrapper[4771]: I1011 10:43:39.629222 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:39.629485 master-1 kubenswrapper[4771]: I1011 10:43:39.629343 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:44.628256 master-1 kubenswrapper[4771]: I1011 10:43:44.628178 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:44.629443 master-1 kubenswrapper[4771]: I1011 10:43:44.628632 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:45.259653 master-1 kubenswrapper[4771]: I1011 10:43:45.259595 4771 kubelet.go:2431] "SyncLoop REMOVE" source="file" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:43:45.260399 master-1 kubenswrapper[4771]: I1011 10:43:45.260296 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager" containerID="cri-o://9e6a4086932c3b4c0590b1992411e46984c974a11450de3378bede5ca3045d02" gracePeriod=30 Oct 11 10:43:45.260634 master-1 kubenswrapper[4771]: I1011 10:43:45.260480 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager-cert-syncer" containerID="cri-o://79e52bbf7393881dfbba04f7a9f71721266d98f1191a6c7be91f8bc0ce4e1139" gracePeriod=30 Oct 11 10:43:45.260730 master-1 kubenswrapper[4771]: I1011 10:43:45.260580 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="cluster-policy-controller" containerID="cri-o://913e0c188082961ad93b5f6a07d9eda57e62160ccbff129947e77948c758035a" gracePeriod=30 Oct 11 10:43:45.262063 master-1 kubenswrapper[4771]: I1011 10:43:45.261391 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:43:45.262063 master-1 kubenswrapper[4771]: E1011 10:43:45.261816 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="cluster-policy-controller" Oct 11 10:43:45.262063 master-1 kubenswrapper[4771]: I1011 10:43:45.261842 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="cluster-policy-controller" Oct 11 10:43:45.262063 master-1 kubenswrapper[4771]: E1011 10:43:45.261860 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager-cert-syncer" Oct 11 10:43:45.262063 master-1 kubenswrapper[4771]: I1011 10:43:45.261873 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager-cert-syncer" Oct 11 10:43:45.262063 master-1 kubenswrapper[4771]: E1011 10:43:45.261893 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager" Oct 11 10:43:45.262063 master-1 kubenswrapper[4771]: I1011 10:43:45.261907 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager" Oct 11 10:43:45.262063 master-1 kubenswrapper[4771]: E1011 10:43:45.261923 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager-recovery-controller" Oct 11 10:43:45.262063 master-1 kubenswrapper[4771]: I1011 10:43:45.261936 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager-recovery-controller" Oct 11 10:43:45.263891 master-1 kubenswrapper[4771]: I1011 10:43:45.262098 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="cluster-policy-controller" Oct 11 10:43:45.263891 master-1 kubenswrapper[4771]: I1011 10:43:45.262121 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager-recovery-controller" Oct 11 10:43:45.263891 master-1 kubenswrapper[4771]: I1011 10:43:45.262143 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager-cert-syncer" Oct 11 10:43:45.263891 master-1 kubenswrapper[4771]: I1011 10:43:45.262165 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager" Oct 11 10:43:45.263891 master-1 kubenswrapper[4771]: I1011 10:43:45.260547 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" containerName="kube-controller-manager-recovery-controller" containerID="cri-o://068b46162b2804f4e661290cc4e58111faa3ee64a5ff733b8a30de9f4b7d070e" gracePeriod=30 Oct 11 10:43:45.433555 master-1 kubenswrapper[4771]: I1011 10:43:45.433482 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/6e4abd751079f7c12d9e1207e209976a-cert-dir\") pod \"kube-controller-manager-master-1\" (UID: \"6e4abd751079f7c12d9e1207e209976a\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:45.433787 master-1 kubenswrapper[4771]: I1011 10:43:45.433568 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/6e4abd751079f7c12d9e1207e209976a-resource-dir\") pod \"kube-controller-manager-master-1\" (UID: \"6e4abd751079f7c12d9e1207e209976a\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:45.459324 master-1 kubenswrapper[4771]: I1011 10:43:45.459026 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_kube-controller-manager-master-1_0b5de9d609ee1e6c379f71934cb2c3c6/kube-controller-manager-cert-syncer/0.log" Oct 11 10:43:45.460214 master-1 kubenswrapper[4771]: I1011 10:43:45.460175 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:45.465933 master-1 kubenswrapper[4771]: I1011 10:43:45.465898 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" oldPodUID="0b5de9d609ee1e6c379f71934cb2c3c6" podUID="6e4abd751079f7c12d9e1207e209976a" Oct 11 10:43:45.535665 master-1 kubenswrapper[4771]: I1011 10:43:45.535600 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-cert-dir\") pod \"0b5de9d609ee1e6c379f71934cb2c3c6\" (UID: \"0b5de9d609ee1e6c379f71934cb2c3c6\") " Oct 11 10:43:45.535875 master-1 kubenswrapper[4771]: I1011 10:43:45.535725 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-cert-dir" (OuterVolumeSpecName: "cert-dir") pod "0b5de9d609ee1e6c379f71934cb2c3c6" (UID: "0b5de9d609ee1e6c379f71934cb2c3c6"). InnerVolumeSpecName "cert-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:45.536093 master-1 kubenswrapper[4771]: I1011 10:43:45.536047 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/6e4abd751079f7c12d9e1207e209976a-cert-dir\") pod \"kube-controller-manager-master-1\" (UID: \"6e4abd751079f7c12d9e1207e209976a\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:45.536407 master-1 kubenswrapper[4771]: I1011 10:43:45.536313 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/6e4abd751079f7c12d9e1207e209976a-cert-dir\") pod \"kube-controller-manager-master-1\" (UID: \"6e4abd751079f7c12d9e1207e209976a\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:45.536854 master-1 kubenswrapper[4771]: I1011 10:43:45.536805 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/6e4abd751079f7c12d9e1207e209976a-resource-dir\") pod \"kube-controller-manager-master-1\" (UID: \"6e4abd751079f7c12d9e1207e209976a\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:45.537068 master-1 kubenswrapper[4771]: I1011 10:43:45.537031 4771 reconciler_common.go:293] "Volume detached for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-cert-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:45.537228 master-1 kubenswrapper[4771]: I1011 10:43:45.537173 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/6e4abd751079f7c12d9e1207e209976a-resource-dir\") pod \"kube-controller-manager-master-1\" (UID: \"6e4abd751079f7c12d9e1207e209976a\") " pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:45.637404 master-1 kubenswrapper[4771]: I1011 10:43:45.637303 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-resource-dir\") pod \"0b5de9d609ee1e6c379f71934cb2c3c6\" (UID: \"0b5de9d609ee1e6c379f71934cb2c3c6\") " Oct 11 10:43:45.638232 master-1 kubenswrapper[4771]: I1011 10:43:45.637509 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-resource-dir" (OuterVolumeSpecName: "resource-dir") pod "0b5de9d609ee1e6c379f71934cb2c3c6" (UID: "0b5de9d609ee1e6c379f71934cb2c3c6"). InnerVolumeSpecName "resource-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:45.638564 master-1 kubenswrapper[4771]: I1011 10:43:45.638543 4771 reconciler_common.go:293] "Volume detached for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/0b5de9d609ee1e6c379f71934cb2c3c6-resource-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:46.304725 master-1 kubenswrapper[4771]: I1011 10:43:46.304655 4771 generic.go:334] "Generic (PLEG): container finished" podID="78689bdc-0258-45eb-8e5b-253911c61c79" containerID="2d907b9a8cd0470d88178cfea01b0abf30291128bc9c158e361b094caee83ec4" exitCode=0 Oct 11 10:43:46.304969 master-1 kubenswrapper[4771]: I1011 10:43:46.304805 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-6-master-1" event={"ID":"78689bdc-0258-45eb-8e5b-253911c61c79","Type":"ContainerDied","Data":"2d907b9a8cd0470d88178cfea01b0abf30291128bc9c158e361b094caee83ec4"} Oct 11 10:43:46.308437 master-1 kubenswrapper[4771]: I1011 10:43:46.308413 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-controller-manager_kube-controller-manager-master-1_0b5de9d609ee1e6c379f71934cb2c3c6/kube-controller-manager-cert-syncer/0.log" Oct 11 10:43:46.309814 master-1 kubenswrapper[4771]: I1011 10:43:46.309781 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b5de9d609ee1e6c379f71934cb2c3c6" containerID="068b46162b2804f4e661290cc4e58111faa3ee64a5ff733b8a30de9f4b7d070e" exitCode=0 Oct 11 10:43:46.309814 master-1 kubenswrapper[4771]: I1011 10:43:46.309811 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b5de9d609ee1e6c379f71934cb2c3c6" containerID="79e52bbf7393881dfbba04f7a9f71721266d98f1191a6c7be91f8bc0ce4e1139" exitCode=2 Oct 11 10:43:46.309932 master-1 kubenswrapper[4771]: I1011 10:43:46.309828 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b5de9d609ee1e6c379f71934cb2c3c6" containerID="913e0c188082961ad93b5f6a07d9eda57e62160ccbff129947e77948c758035a" exitCode=0 Oct 11 10:43:46.309932 master-1 kubenswrapper[4771]: I1011 10:43:46.309842 4771 generic.go:334] "Generic (PLEG): container finished" podID="0b5de9d609ee1e6c379f71934cb2c3c6" containerID="9e6a4086932c3b4c0590b1992411e46984c974a11450de3378bede5ca3045d02" exitCode=0 Oct 11 10:43:46.309932 master-1 kubenswrapper[4771]: I1011 10:43:46.309889 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4247c914a32e821feeb321db49e7b5b061a40ecb112a752686b9ea07098f462f" Oct 11 10:43:46.309932 master-1 kubenswrapper[4771]: I1011 10:43:46.309912 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:46.331115 master-1 kubenswrapper[4771]: I1011 10:43:46.331050 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" oldPodUID="0b5de9d609ee1e6c379f71934cb2c3c6" podUID="6e4abd751079f7c12d9e1207e209976a" Oct 11 10:43:46.339983 master-1 kubenswrapper[4771]: I1011 10:43:46.339925 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" oldPodUID="0b5de9d609ee1e6c379f71934cb2c3c6" podUID="6e4abd751079f7c12d9e1207e209976a" Oct 11 10:43:46.436933 master-1 kubenswrapper[4771]: I1011 10:43:46.436860 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:43:46.448100 master-1 kubenswrapper[4771]: I1011 10:43:46.448048 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5de9d609ee1e6c379f71934cb2c3c6" path="/var/lib/kubelet/pods/0b5de9d609ee1e6c379f71934cb2c3c6/volumes" Oct 11 10:43:46.452174 master-1 kubenswrapper[4771]: I1011 10:43:46.452134 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-etcd/etcd-master-1" podUID="6c6a59a5-a8d4-44d0-bc98-101ffe6273e9" Oct 11 10:43:46.452232 master-1 kubenswrapper[4771]: I1011 10:43:46.452174 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-etcd/etcd-master-1" podUID="6c6a59a5-a8d4-44d0-bc98-101ffe6273e9" Oct 11 10:43:46.473876 master-1 kubenswrapper[4771]: I1011 10:43:46.473824 4771 kubelet.go:1914] "Deleted mirror pod because it is outdated" pod="openshift-etcd/etcd-master-1" Oct 11 10:43:46.474212 master-1 kubenswrapper[4771]: I1011 10:43:46.474188 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:43:46.482370 master-1 kubenswrapper[4771]: I1011 10:43:46.482297 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:43:46.499243 master-1 kubenswrapper[4771]: I1011 10:43:46.498507 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/etcd-master-1" Oct 11 10:43:46.501607 master-1 kubenswrapper[4771]: I1011 10:43:46.501556 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-etcd/etcd-master-1"] Oct 11 10:43:46.523008 master-1 kubenswrapper[4771]: W1011 10:43:46.522929 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbeb1098f6b7e52b91afcf2e9b50b014.slice/crio-85ffa4a19feba0ffcf516a2a5e009b706621b05b01a7558d41fc981463ac60e2 WatchSource:0}: Error finding container 85ffa4a19feba0ffcf516a2a5e009b706621b05b01a7558d41fc981463ac60e2: Status 404 returned error can't find the container with id 85ffa4a19feba0ffcf516a2a5e009b706621b05b01a7558d41fc981463ac60e2 Oct 11 10:43:47.294167 master-1 kubenswrapper[4771]: I1011 10:43:47.293918 4771 patch_prober.go:28] interesting pod/kube-controller-manager-guard-master-1 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" start-of-body= Oct 11 10:43:47.294167 master-1 kubenswrapper[4771]: I1011 10:43:47.294064 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podUID="a706deec-9223-4663-9db5-71147d242c34" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" Oct 11 10:43:47.321457 master-1 kubenswrapper[4771]: I1011 10:43:47.321335 4771 generic.go:334] "Generic (PLEG): container finished" podID="dbeb1098f6b7e52b91afcf2e9b50b014" containerID="211daec19e26fca55ad8690f95b0fed282fad8cd036efbb54c03ad1969a7cfb2" exitCode=0 Oct 11 10:43:47.321599 master-1 kubenswrapper[4771]: I1011 10:43:47.321499 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"dbeb1098f6b7e52b91afcf2e9b50b014","Type":"ContainerDied","Data":"211daec19e26fca55ad8690f95b0fed282fad8cd036efbb54c03ad1969a7cfb2"} Oct 11 10:43:47.321599 master-1 kubenswrapper[4771]: I1011 10:43:47.321568 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"dbeb1098f6b7e52b91afcf2e9b50b014","Type":"ContainerStarted","Data":"85ffa4a19feba0ffcf516a2a5e009b706621b05b01a7558d41fc981463ac60e2"} Oct 11 10:43:47.752968 master-1 kubenswrapper[4771]: I1011 10:43:47.752876 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:47.874886 master-1 kubenswrapper[4771]: I1011 10:43:47.874670 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/78689bdc-0258-45eb-8e5b-253911c61c79-kube-api-access\") pod \"78689bdc-0258-45eb-8e5b-253911c61c79\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " Oct 11 10:43:47.874886 master-1 kubenswrapper[4771]: I1011 10:43:47.874845 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-kubelet-dir\") pod \"78689bdc-0258-45eb-8e5b-253911c61c79\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " Oct 11 10:43:47.875284 master-1 kubenswrapper[4771]: I1011 10:43:47.874985 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "78689bdc-0258-45eb-8e5b-253911c61c79" (UID: "78689bdc-0258-45eb-8e5b-253911c61c79"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:47.875284 master-1 kubenswrapper[4771]: I1011 10:43:47.875111 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-var-lock\") pod \"78689bdc-0258-45eb-8e5b-253911c61c79\" (UID: \"78689bdc-0258-45eb-8e5b-253911c61c79\") " Oct 11 10:43:47.875284 master-1 kubenswrapper[4771]: I1011 10:43:47.875203 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-var-lock" (OuterVolumeSpecName: "var-lock") pod "78689bdc-0258-45eb-8e5b-253911c61c79" (UID: "78689bdc-0258-45eb-8e5b-253911c61c79"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:43:47.875657 master-1 kubenswrapper[4771]: I1011 10:43:47.875566 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:47.875657 master-1 kubenswrapper[4771]: I1011 10:43:47.875623 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/78689bdc-0258-45eb-8e5b-253911c61c79-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:47.880725 master-1 kubenswrapper[4771]: I1011 10:43:47.880619 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78689bdc-0258-45eb-8e5b-253911c61c79-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "78689bdc-0258-45eb-8e5b-253911c61c79" (UID: "78689bdc-0258-45eb-8e5b-253911c61c79"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:43:47.976924 master-1 kubenswrapper[4771]: I1011 10:43:47.976832 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/78689bdc-0258-45eb-8e5b-253911c61c79-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:43:48.341287 master-1 kubenswrapper[4771]: I1011 10:43:48.341214 4771 generic.go:334] "Generic (PLEG): container finished" podID="dbeb1098f6b7e52b91afcf2e9b50b014" containerID="31facdd8fb6e6f6274c00fada32ab1255ea6776f85ffbc3f8065c95c2d2382fb" exitCode=0 Oct 11 10:43:48.342111 master-1 kubenswrapper[4771]: I1011 10:43:48.341283 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"dbeb1098f6b7e52b91afcf2e9b50b014","Type":"ContainerDied","Data":"31facdd8fb6e6f6274c00fada32ab1255ea6776f85ffbc3f8065c95c2d2382fb"} Oct 11 10:43:48.345338 master-1 kubenswrapper[4771]: I1011 10:43:48.345251 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/installer-6-master-1" event={"ID":"78689bdc-0258-45eb-8e5b-253911c61c79","Type":"ContainerDied","Data":"56046807ca0aa308b5de3a340c0e367a86144d71f324a1b35aa3b72ae51dc090"} Oct 11 10:43:48.345476 master-1 kubenswrapper[4771]: I1011 10:43:48.345288 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/installer-6-master-1" Oct 11 10:43:48.345542 master-1 kubenswrapper[4771]: I1011 10:43:48.345389 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56046807ca0aa308b5de3a340c0e367a86144d71f324a1b35aa3b72ae51dc090" Oct 11 10:43:49.357612 master-1 kubenswrapper[4771]: I1011 10:43:49.357551 4771 generic.go:334] "Generic (PLEG): container finished" podID="dbeb1098f6b7e52b91afcf2e9b50b014" containerID="159ca90da6e99bed0d178155a8c50681923c5b6021a8639d479924886947bb47" exitCode=0 Oct 11 10:43:49.358544 master-1 kubenswrapper[4771]: I1011 10:43:49.357648 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"dbeb1098f6b7e52b91afcf2e9b50b014","Type":"ContainerDied","Data":"159ca90da6e99bed0d178155a8c50681923c5b6021a8639d479924886947bb47"} Oct 11 10:43:49.629225 master-1 kubenswrapper[4771]: I1011 10:43:49.629111 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" start-of-body= Oct 11 10:43:49.629423 master-1 kubenswrapper[4771]: I1011 10:43:49.629235 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": dial tcp 192.168.34.11:9980: connect: connection refused" Oct 11 10:43:50.375343 master-1 kubenswrapper[4771]: I1011 10:43:50.375268 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"dbeb1098f6b7e52b91afcf2e9b50b014","Type":"ContainerStarted","Data":"9a89c09ca6e91647da081f97087658c1b11fe705a16ff46043003c3fbbcd0e8e"} Oct 11 10:43:50.375343 master-1 kubenswrapper[4771]: I1011 10:43:50.375340 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"dbeb1098f6b7e52b91afcf2e9b50b014","Type":"ContainerStarted","Data":"6d52caa6ca3071165a980d0d3715ea1f4a335edb9977e20dd52616ba6ac3305d"} Oct 11 10:43:51.391239 master-1 kubenswrapper[4771]: I1011 10:43:51.391162 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"dbeb1098f6b7e52b91afcf2e9b50b014","Type":"ContainerStarted","Data":"2230bc9413b16f3b0764a9834ddc735459529336c9441bb2965aa9ffe5d841d9"} Oct 11 10:43:51.391239 master-1 kubenswrapper[4771]: I1011 10:43:51.391247 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"dbeb1098f6b7e52b91afcf2e9b50b014","Type":"ContainerStarted","Data":"269a69a20048205c2f87c7a43fc6a19fb199854bc247ee4dfd460bbf2b358b62"} Oct 11 10:43:51.392128 master-1 kubenswrapper[4771]: I1011 10:43:51.391268 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/etcd-master-1" event={"ID":"dbeb1098f6b7e52b91afcf2e9b50b014","Type":"ContainerStarted","Data":"c78b119fc172933815c1d2c54198af0cd4a86318e7a1301b7b416c3fad42949a"} Oct 11 10:43:51.448024 master-1 kubenswrapper[4771]: I1011 10:43:51.447946 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-etcd/etcd-master-1" podStartSLOduration=5.447926073 podStartE2EDuration="5.447926073s" podCreationTimestamp="2025-10-11 10:43:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:43:51.444532446 +0000 UTC m=+1063.418758917" watchObservedRunningTime="2025-10-11 10:43:51.447926073 +0000 UTC m=+1063.422152514" Oct 11 10:43:51.499767 master-1 kubenswrapper[4771]: I1011 10:43:51.499679 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-etcd/etcd-master-1" Oct 11 10:43:52.293978 master-1 kubenswrapper[4771]: I1011 10:43:52.293883 4771 patch_prober.go:28] interesting pod/kube-controller-manager-guard-master-1 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" start-of-body= Oct 11 10:43:52.294205 master-1 kubenswrapper[4771]: I1011 10:43:52.293983 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podUID="a706deec-9223-4663-9db5-71147d242c34" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" Oct 11 10:43:56.499394 master-1 kubenswrapper[4771]: I1011 10:43:56.499278 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-etcd/etcd-master-1" Oct 11 10:43:57.294618 master-1 kubenswrapper[4771]: I1011 10:43:57.294524 4771 patch_prober.go:28] interesting pod/kube-controller-manager-guard-master-1 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" start-of-body= Oct 11 10:43:57.294919 master-1 kubenswrapper[4771]: I1011 10:43:57.294711 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podUID="a706deec-9223-4663-9db5-71147d242c34" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" Oct 11 10:43:57.294919 master-1 kubenswrapper[4771]: I1011 10:43:57.294863 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:43:57.296038 master-1 kubenswrapper[4771]: I1011 10:43:57.295968 4771 patch_prober.go:28] interesting pod/kube-controller-manager-guard-master-1 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" start-of-body= Oct 11 10:43:57.296131 master-1 kubenswrapper[4771]: I1011 10:43:57.296061 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" podUID="a706deec-9223-4663-9db5-71147d242c34" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:10257/healthz\": dial tcp 192.168.34.11:10257: connect: connection refused" Oct 11 10:43:59.437167 master-1 kubenswrapper[4771]: I1011 10:43:59.437053 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:59.453973 master-1 kubenswrapper[4771]: I1011 10:43:59.453903 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="6daea3ba-d094-4b54-9863-c829a9c42066" Oct 11 10:43:59.453973 master-1 kubenswrapper[4771]: I1011 10:43:59.453960 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podUID="6daea3ba-d094-4b54-9863-c829a9c42066" Oct 11 10:43:59.477804 master-1 kubenswrapper[4771]: I1011 10:43:59.477414 4771 kubelet.go:1914] "Deleted mirror pod because it is outdated" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:59.479725 master-1 kubenswrapper[4771]: I1011 10:43:59.477820 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:43:59.486225 master-1 kubenswrapper[4771]: I1011 10:43:59.486138 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:43:59.497456 master-1 kubenswrapper[4771]: I1011 10:43:59.497295 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:43:59.503923 master-1 kubenswrapper[4771]: I1011 10:43:59.503815 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-controller-manager/kube-controller-manager-master-1"] Oct 11 10:43:59.530568 master-1 kubenswrapper[4771]: W1011 10:43:59.530488 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4abd751079f7c12d9e1207e209976a.slice/crio-3a658d0c2cfd0d7018a054674ad2fef1705a690b9977d860ea43a5927f7570a2 WatchSource:0}: Error finding container 3a658d0c2cfd0d7018a054674ad2fef1705a690b9977d860ea43a5927f7570a2: Status 404 returned error can't find the container with id 3a658d0c2cfd0d7018a054674ad2fef1705a690b9977d860ea43a5927f7570a2 Oct 11 10:43:59.629407 master-1 kubenswrapper[4771]: I1011 10:43:59.629313 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:43:59.629759 master-1 kubenswrapper[4771]: I1011 10:43:59.629718 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:00.459888 master-1 kubenswrapper[4771]: I1011 10:44:00.459828 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"6e4abd751079f7c12d9e1207e209976a","Type":"ContainerStarted","Data":"2b8cfa34057f1d7441fa8899d6ffb269449e64e8f7fbad7e041afde233f9f46e"} Oct 11 10:44:00.460384 master-1 kubenswrapper[4771]: I1011 10:44:00.459902 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"6e4abd751079f7c12d9e1207e209976a","Type":"ContainerStarted","Data":"47f67f1edcd45fc05bb0497d766cd846facb4877f08282bc158671c2139f4123"} Oct 11 10:44:00.460384 master-1 kubenswrapper[4771]: I1011 10:44:00.459926 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"6e4abd751079f7c12d9e1207e209976a","Type":"ContainerStarted","Data":"698e3875aa3788d3ef1c91df166ebdd80850c7ea07b48f8abbf75cab790053eb"} Oct 11 10:44:00.460384 master-1 kubenswrapper[4771]: I1011 10:44:00.459946 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"6e4abd751079f7c12d9e1207e209976a","Type":"ContainerStarted","Data":"3a658d0c2cfd0d7018a054674ad2fef1705a690b9977d860ea43a5927f7570a2"} Oct 11 10:44:01.471522 master-1 kubenswrapper[4771]: I1011 10:44:01.471449 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" event={"ID":"6e4abd751079f7c12d9e1207e209976a","Type":"ContainerStarted","Data":"58932b681106833dbc88897b7539419e139110df78331149ef2988d73886577e"} Oct 11 10:44:01.498876 master-1 kubenswrapper[4771]: I1011 10:44:01.498781 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" podStartSLOduration=2.498758567 podStartE2EDuration="2.498758567s" podCreationTimestamp="2025-10-11 10:43:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:44:01.497206483 +0000 UTC m=+1073.471432964" watchObservedRunningTime="2025-10-11 10:44:01.498758567 +0000 UTC m=+1073.472985048" Oct 11 10:44:02.299850 master-1 kubenswrapper[4771]: I1011 10:44:02.299765 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-1" Oct 11 10:44:04.631089 master-1 kubenswrapper[4771]: I1011 10:44:04.631009 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:04.631089 master-1 kubenswrapper[4771]: I1011 10:44:04.631076 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:07.500376 master-1 kubenswrapper[4771]: I1011 10:44:07.500290 4771 patch_prober.go:28] interesting pod/etcd-master-1 container/etcd namespace/openshift-etcd: Startup probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:07.500993 master-1 kubenswrapper[4771]: I1011 10:44:07.500385 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-etcd/etcd-master-1" podUID="dbeb1098f6b7e52b91afcf2e9b50b014" containerName="etcd" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:09.498403 master-1 kubenswrapper[4771]: I1011 10:44:09.498286 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:44:09.498403 master-1 kubenswrapper[4771]: I1011 10:44:09.498405 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:44:09.499878 master-1 kubenswrapper[4771]: I1011 10:44:09.498434 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:44:09.499878 master-1 kubenswrapper[4771]: I1011 10:44:09.498455 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:44:09.504683 master-1 kubenswrapper[4771]: I1011 10:44:09.504624 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:44:09.506477 master-1 kubenswrapper[4771]: I1011 10:44:09.506436 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:44:09.543338 master-1 kubenswrapper[4771]: I1011 10:44:09.543266 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:44:09.632029 master-1 kubenswrapper[4771]: I1011 10:44:09.631899 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:09.632029 master-1 kubenswrapper[4771]: I1011 10:44:09.632022 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:10.550064 master-1 kubenswrapper[4771]: I1011 10:44:10.549978 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-master-1" Oct 11 10:44:14.632421 master-1 kubenswrapper[4771]: I1011 10:44:14.632298 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:14.633341 master-1 kubenswrapper[4771]: I1011 10:44:14.632428 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:17.500682 master-1 kubenswrapper[4771]: I1011 10:44:17.500599 4771 patch_prober.go:28] interesting pod/etcd-master-1 container/etcd namespace/openshift-etcd: Startup probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:17.501709 master-1 kubenswrapper[4771]: I1011 10:44:17.500703 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-etcd/etcd-master-1" podUID="dbeb1098f6b7e52b91afcf2e9b50b014" containerName="etcd" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:19.638857 master-1 kubenswrapper[4771]: I1011 10:44:19.634524 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:19.638857 master-1 kubenswrapper[4771]: I1011 10:44:19.635471 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:24.636924 master-1 kubenswrapper[4771]: I1011 10:44:24.636769 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:24.636924 master-1 kubenswrapper[4771]: I1011 10:44:24.636917 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:27.500193 master-1 kubenswrapper[4771]: I1011 10:44:27.500099 4771 patch_prober.go:28] interesting pod/etcd-master-1 container/etcd namespace/openshift-etcd: Startup probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:27.500801 master-1 kubenswrapper[4771]: I1011 10:44:27.500189 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-etcd/etcd-master-1" podUID="dbeb1098f6b7e52b91afcf2e9b50b014" containerName="etcd" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:29.637677 master-1 kubenswrapper[4771]: I1011 10:44:29.637559 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:29.638731 master-1 kubenswrapper[4771]: I1011 10:44:29.637703 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:34.638238 master-1 kubenswrapper[4771]: I1011 10:44:34.638132 4771 patch_prober.go:28] interesting pod/etcd-guard-master-1 container/guard namespace/openshift-etcd: Readiness probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:34.639591 master-1 kubenswrapper[4771]: I1011 10:44:34.638270 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-etcd/etcd-guard-master-1" podUID="3fc4970d-4f34-4fc6-9791-6218f8e42eb9" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:37.500910 master-1 kubenswrapper[4771]: I1011 10:44:37.500791 4771 patch_prober.go:28] interesting pod/etcd-master-1 container/etcd namespace/openshift-etcd: Startup probe status=failure output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 11 10:44:37.500910 master-1 kubenswrapper[4771]: I1011 10:44:37.500878 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-etcd/etcd-master-1" podUID="dbeb1098f6b7e52b91afcf2e9b50b014" containerName="etcd" probeResult="failure" output="Get \"https://192.168.34.11:9980/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:44:37.830847 master-1 kubenswrapper[4771]: I1011 10:44:37.830754 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b846b7bb4-xmv6l"] Oct 11 10:44:39.316416 master-1 kubenswrapper[4771]: I1011 10:44:39.316319 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-etcd/etcd-guard-master-1" Oct 11 10:44:46.518968 master-1 kubenswrapper[4771]: I1011 10:44:46.518894 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-etcd/etcd-master-1" Oct 11 10:44:46.538129 master-1 kubenswrapper[4771]: I1011 10:44:46.538019 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-etcd/etcd-master-1" Oct 11 10:44:59.403085 master-1 kubenswrapper[4771]: I1011 10:44:59.402913 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-operators-fn27x"] Oct 11 10:44:59.404566 master-1 kubenswrapper[4771]: E1011 10:44:59.403301 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="78689bdc-0258-45eb-8e5b-253911c61c79" containerName="installer" Oct 11 10:44:59.404566 master-1 kubenswrapper[4771]: I1011 10:44:59.403322 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="78689bdc-0258-45eb-8e5b-253911c61c79" containerName="installer" Oct 11 10:44:59.404566 master-1 kubenswrapper[4771]: I1011 10:44:59.403568 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="78689bdc-0258-45eb-8e5b-253911c61c79" containerName="installer" Oct 11 10:44:59.413032 master-1 kubenswrapper[4771]: I1011 10:44:59.412913 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.429551 master-1 kubenswrapper[4771]: I1011 10:44:59.428588 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-fn27x"] Oct 11 10:44:59.442102 master-1 kubenswrapper[4771]: I1011 10:44:59.441930 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-utilities\") pod \"redhat-operators-fn27x\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.442102 master-1 kubenswrapper[4771]: I1011 10:44:59.442013 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-catalog-content\") pod \"redhat-operators-fn27x\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.442102 master-1 kubenswrapper[4771]: I1011 10:44:59.442060 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8rx\" (UniqueName: \"kubernetes.io/projected/dd28168d-b375-4a82-8784-bc38fad4cc07-kube-api-access-4z8rx\") pod \"redhat-operators-fn27x\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.543672 master-1 kubenswrapper[4771]: I1011 10:44:59.543586 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-utilities\") pod \"redhat-operators-fn27x\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.544027 master-1 kubenswrapper[4771]: I1011 10:44:59.543690 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-catalog-content\") pod \"redhat-operators-fn27x\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.544027 master-1 kubenswrapper[4771]: I1011 10:44:59.543771 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8rx\" (UniqueName: \"kubernetes.io/projected/dd28168d-b375-4a82-8784-bc38fad4cc07-kube-api-access-4z8rx\") pod \"redhat-operators-fn27x\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.546445 master-1 kubenswrapper[4771]: I1011 10:44:59.546378 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-utilities\") pod \"redhat-operators-fn27x\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.546964 master-1 kubenswrapper[4771]: I1011 10:44:59.546917 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-catalog-content\") pod \"redhat-operators-fn27x\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.577526 master-1 kubenswrapper[4771]: I1011 10:44:59.575766 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8rx\" (UniqueName: \"kubernetes.io/projected/dd28168d-b375-4a82-8784-bc38fad4cc07-kube-api-access-4z8rx\") pod \"redhat-operators-fn27x\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.602008 master-1 kubenswrapper[4771]: I1011 10:44:59.601889 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-marketplace-btlwb"] Oct 11 10:44:59.603807 master-1 kubenswrapper[4771]: I1011 10:44:59.603759 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.617627 master-1 kubenswrapper[4771]: I1011 10:44:59.617540 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-btlwb"] Oct 11 10:44:59.645461 master-1 kubenswrapper[4771]: I1011 10:44:59.645046 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-utilities\") pod \"redhat-marketplace-btlwb\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.645461 master-1 kubenswrapper[4771]: I1011 10:44:59.645146 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-catalog-content\") pod \"redhat-marketplace-btlwb\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.645461 master-1 kubenswrapper[4771]: I1011 10:44:59.645183 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjs9q\" (UniqueName: \"kubernetes.io/projected/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-kube-api-access-jjs9q\") pod \"redhat-marketplace-btlwb\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.746771 master-1 kubenswrapper[4771]: I1011 10:44:59.746615 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-catalog-content\") pod \"redhat-marketplace-btlwb\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.746771 master-1 kubenswrapper[4771]: I1011 10:44:59.746695 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-jjs9q\" (UniqueName: \"kubernetes.io/projected/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-kube-api-access-jjs9q\") pod \"redhat-marketplace-btlwb\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.746771 master-1 kubenswrapper[4771]: I1011 10:44:59.746743 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-utilities\") pod \"redhat-marketplace-btlwb\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.747465 master-1 kubenswrapper[4771]: I1011 10:44:59.747408 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-catalog-content\") pod \"redhat-marketplace-btlwb\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.747546 master-1 kubenswrapper[4771]: I1011 10:44:59.747487 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-utilities\") pod \"redhat-marketplace-btlwb\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.754227 master-1 kubenswrapper[4771]: I1011 10:44:59.754182 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:44:59.773736 master-1 kubenswrapper[4771]: I1011 10:44:59.773582 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjs9q\" (UniqueName: \"kubernetes.io/projected/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-kube-api-access-jjs9q\") pod \"redhat-marketplace-btlwb\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:44:59.951808 master-1 kubenswrapper[4771]: I1011 10:44:59.949992 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:45:00.040476 master-1 kubenswrapper[4771]: I1011 10:45:00.038667 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-fn27x"] Oct 11 10:45:00.385116 master-1 kubenswrapper[4771]: I1011 10:45:00.385057 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-btlwb"] Oct 11 10:45:00.390059 master-1 kubenswrapper[4771]: W1011 10:45:00.390014 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7fddf6_d341_4992_bba8_9d5fa5b1e7a1.slice/crio-07f0178441f3d50314b9519115a4f3d7e321d0618d45e6c54bc9d3b3f3be9ea8 WatchSource:0}: Error finding container 07f0178441f3d50314b9519115a4f3d7e321d0618d45e6c54bc9d3b3f3be9ea8: Status 404 returned error can't find the container with id 07f0178441f3d50314b9519115a4f3d7e321d0618d45e6c54bc9d3b3f3be9ea8 Oct 11 10:45:00.926303 master-1 kubenswrapper[4771]: I1011 10:45:00.926233 4771 generic.go:334] "Generic (PLEG): container finished" podID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerID="563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245" exitCode=0 Oct 11 10:45:00.926303 master-1 kubenswrapper[4771]: I1011 10:45:00.926313 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-btlwb" event={"ID":"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1","Type":"ContainerDied","Data":"563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245"} Oct 11 10:45:00.927161 master-1 kubenswrapper[4771]: I1011 10:45:00.926341 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-btlwb" event={"ID":"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1","Type":"ContainerStarted","Data":"07f0178441f3d50314b9519115a4f3d7e321d0618d45e6c54bc9d3b3f3be9ea8"} Oct 11 10:45:00.927614 master-1 kubenswrapper[4771]: I1011 10:45:00.927596 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 10:45:00.927933 master-1 kubenswrapper[4771]: I1011 10:45:00.927895 4771 generic.go:334] "Generic (PLEG): container finished" podID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerID="09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132" exitCode=0 Oct 11 10:45:00.927933 master-1 kubenswrapper[4771]: I1011 10:45:00.927923 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-fn27x" event={"ID":"dd28168d-b375-4a82-8784-bc38fad4cc07","Type":"ContainerDied","Data":"09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132"} Oct 11 10:45:00.928029 master-1 kubenswrapper[4771]: I1011 10:45:00.927939 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-fn27x" event={"ID":"dd28168d-b375-4a82-8784-bc38fad4cc07","Type":"ContainerStarted","Data":"6a5c5b9ff534c3108ab033d2a189e452b9f1cc8c2fa78f601306eadfc2f6563e"} Oct 11 10:45:01.938342 master-1 kubenswrapper[4771]: I1011 10:45:01.937632 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-fn27x" event={"ID":"dd28168d-b375-4a82-8784-bc38fad4cc07","Type":"ContainerStarted","Data":"63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8"} Oct 11 10:45:02.003079 master-1 kubenswrapper[4771]: I1011 10:45:02.002933 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/community-operators-r8hdr"] Oct 11 10:45:02.005198 master-1 kubenswrapper[4771]: I1011 10:45:02.005142 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.024524 master-1 kubenswrapper[4771]: I1011 10:45:02.024469 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-r8hdr"] Oct 11 10:45:02.083701 master-1 kubenswrapper[4771]: I1011 10:45:02.083624 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plx55\" (UniqueName: \"kubernetes.io/projected/8e964e77-4315-44b2-a34f-d0e2249e9a72-kube-api-access-plx55\") pod \"community-operators-r8hdr\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.084036 master-1 kubenswrapper[4771]: I1011 10:45:02.083740 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-utilities\") pod \"community-operators-r8hdr\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.084036 master-1 kubenswrapper[4771]: I1011 10:45:02.083788 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-catalog-content\") pod \"community-operators-r8hdr\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.185284 master-1 kubenswrapper[4771]: I1011 10:45:02.185210 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-plx55\" (UniqueName: \"kubernetes.io/projected/8e964e77-4315-44b2-a34f-d0e2249e9a72-kube-api-access-plx55\") pod \"community-operators-r8hdr\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.185553 master-1 kubenswrapper[4771]: I1011 10:45:02.185330 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-utilities\") pod \"community-operators-r8hdr\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.185553 master-1 kubenswrapper[4771]: I1011 10:45:02.185397 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-catalog-content\") pod \"community-operators-r8hdr\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.185965 master-1 kubenswrapper[4771]: I1011 10:45:02.185931 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-utilities\") pod \"community-operators-r8hdr\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.186202 master-1 kubenswrapper[4771]: I1011 10:45:02.186147 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-catalog-content\") pod \"community-operators-r8hdr\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.217964 master-1 kubenswrapper[4771]: I1011 10:45:02.217898 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-plx55\" (UniqueName: \"kubernetes.io/projected/8e964e77-4315-44b2-a34f-d0e2249e9a72-kube-api-access-plx55\") pod \"community-operators-r8hdr\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.328930 master-1 kubenswrapper[4771]: I1011 10:45:02.328825 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:02.790673 master-1 kubenswrapper[4771]: I1011 10:45:02.790618 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-r8hdr"] Oct 11 10:45:02.794818 master-1 kubenswrapper[4771]: W1011 10:45:02.794767 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e964e77_4315_44b2_a34f_d0e2249e9a72.slice/crio-7305b66b7aeba582a9f93dc93062bbbc5bb8eccd416f40fbbfbc9aebdb769b49 WatchSource:0}: Error finding container 7305b66b7aeba582a9f93dc93062bbbc5bb8eccd416f40fbbfbc9aebdb769b49: Status 404 returned error can't find the container with id 7305b66b7aeba582a9f93dc93062bbbc5bb8eccd416f40fbbfbc9aebdb769b49 Oct 11 10:45:02.889252 master-1 kubenswrapper[4771]: I1011 10:45:02.889155 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-console/console-5b846b7bb4-xmv6l" podUID="a65b0165-5747-48c9-9179-86f19861dd68" containerName="console" containerID="cri-o://6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6" gracePeriod=15 Oct 11 10:45:02.945447 master-1 kubenswrapper[4771]: I1011 10:45:02.945396 4771 generic.go:334] "Generic (PLEG): container finished" podID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerID="63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8" exitCode=0 Oct 11 10:45:02.945800 master-1 kubenswrapper[4771]: I1011 10:45:02.945468 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-fn27x" event={"ID":"dd28168d-b375-4a82-8784-bc38fad4cc07","Type":"ContainerDied","Data":"63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8"} Oct 11 10:45:02.948822 master-1 kubenswrapper[4771]: I1011 10:45:02.948790 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-r8hdr" event={"ID":"8e964e77-4315-44b2-a34f-d0e2249e9a72","Type":"ContainerStarted","Data":"7305b66b7aeba582a9f93dc93062bbbc5bb8eccd416f40fbbfbc9aebdb769b49"} Oct 11 10:45:02.950514 master-1 kubenswrapper[4771]: I1011 10:45:02.950484 4771 generic.go:334] "Generic (PLEG): container finished" podID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerID="b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d" exitCode=0 Oct 11 10:45:02.950514 master-1 kubenswrapper[4771]: I1011 10:45:02.950511 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-btlwb" event={"ID":"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1","Type":"ContainerDied","Data":"b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d"} Oct 11 10:45:03.404837 master-1 kubenswrapper[4771]: I1011 10:45:03.404777 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b846b7bb4-xmv6l_a65b0165-5747-48c9-9179-86f19861dd68/console/0.log" Oct 11 10:45:03.404980 master-1 kubenswrapper[4771]: I1011 10:45:03.404856 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:45:03.505694 master-1 kubenswrapper[4771]: I1011 10:45:03.505642 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-console-config\") pod \"a65b0165-5747-48c9-9179-86f19861dd68\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " Oct 11 10:45:03.505867 master-1 kubenswrapper[4771]: I1011 10:45:03.505690 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-service-ca\") pod \"a65b0165-5747-48c9-9179-86f19861dd68\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " Oct 11 10:45:03.505867 master-1 kubenswrapper[4771]: I1011 10:45:03.505742 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpp7h\" (UniqueName: \"kubernetes.io/projected/a65b0165-5747-48c9-9179-86f19861dd68-kube-api-access-qpp7h\") pod \"a65b0165-5747-48c9-9179-86f19861dd68\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " Oct 11 10:45:03.505867 master-1 kubenswrapper[4771]: I1011 10:45:03.505778 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-serving-cert\") pod \"a65b0165-5747-48c9-9179-86f19861dd68\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " Oct 11 10:45:03.506250 master-1 kubenswrapper[4771]: I1011 10:45:03.506216 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-oauth-serving-cert\") pod \"a65b0165-5747-48c9-9179-86f19861dd68\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " Oct 11 10:45:03.506653 master-1 kubenswrapper[4771]: I1011 10:45:03.506250 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-oauth-config\") pod \"a65b0165-5747-48c9-9179-86f19861dd68\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " Oct 11 10:45:03.506696 master-1 kubenswrapper[4771]: I1011 10:45:03.506676 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-trusted-ca-bundle\") pod \"a65b0165-5747-48c9-9179-86f19861dd68\" (UID: \"a65b0165-5747-48c9-9179-86f19861dd68\") " Oct 11 10:45:03.506746 master-1 kubenswrapper[4771]: I1011 10:45:03.506583 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-service-ca" (OuterVolumeSpecName: "service-ca") pod "a65b0165-5747-48c9-9179-86f19861dd68" (UID: "a65b0165-5747-48c9-9179-86f19861dd68"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:45:03.507075 master-1 kubenswrapper[4771]: I1011 10:45:03.507012 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-console-config" (OuterVolumeSpecName: "console-config") pod "a65b0165-5747-48c9-9179-86f19861dd68" (UID: "a65b0165-5747-48c9-9179-86f19861dd68"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:45:03.507294 master-1 kubenswrapper[4771]: I1011 10:45:03.507264 4771 reconciler_common.go:293] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-console-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:03.507294 master-1 kubenswrapper[4771]: I1011 10:45:03.507291 4771 reconciler_common.go:293] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-service-ca\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:03.508074 master-1 kubenswrapper[4771]: I1011 10:45:03.508020 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a65b0165-5747-48c9-9179-86f19861dd68" (UID: "a65b0165-5747-48c9-9179-86f19861dd68"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:45:03.508135 master-1 kubenswrapper[4771]: I1011 10:45:03.508089 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a65b0165-5747-48c9-9179-86f19861dd68" (UID: "a65b0165-5747-48c9-9179-86f19861dd68"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:45:03.510894 master-1 kubenswrapper[4771]: I1011 10:45:03.510837 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65b0165-5747-48c9-9179-86f19861dd68-kube-api-access-qpp7h" (OuterVolumeSpecName: "kube-api-access-qpp7h") pod "a65b0165-5747-48c9-9179-86f19861dd68" (UID: "a65b0165-5747-48c9-9179-86f19861dd68"). InnerVolumeSpecName "kube-api-access-qpp7h". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:45:03.511320 master-1 kubenswrapper[4771]: I1011 10:45:03.511272 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a65b0165-5747-48c9-9179-86f19861dd68" (UID: "a65b0165-5747-48c9-9179-86f19861dd68"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:45:03.511532 master-1 kubenswrapper[4771]: I1011 10:45:03.511498 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a65b0165-5747-48c9-9179-86f19861dd68" (UID: "a65b0165-5747-48c9-9179-86f19861dd68"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:45:03.609606 master-1 kubenswrapper[4771]: I1011 10:45:03.609400 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-qpp7h\" (UniqueName: \"kubernetes.io/projected/a65b0165-5747-48c9-9179-86f19861dd68-kube-api-access-qpp7h\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:03.609606 master-1 kubenswrapper[4771]: I1011 10:45:03.609459 4771 reconciler_common.go:293] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:03.609606 master-1 kubenswrapper[4771]: I1011 10:45:03.609477 4771 reconciler_common.go:293] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-oauth-serving-cert\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:03.609606 master-1 kubenswrapper[4771]: I1011 10:45:03.609492 4771 reconciler_common.go:293] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a65b0165-5747-48c9-9179-86f19861dd68-console-oauth-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:03.609606 master-1 kubenswrapper[4771]: I1011 10:45:03.609503 4771 reconciler_common.go:293] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65b0165-5747-48c9-9179-86f19861dd68-trusted-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:03.961161 master-1 kubenswrapper[4771]: I1011 10:45:03.960977 4771 generic.go:334] "Generic (PLEG): container finished" podID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerID="77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1" exitCode=0 Oct 11 10:45:03.961961 master-1 kubenswrapper[4771]: I1011 10:45:03.961156 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-r8hdr" event={"ID":"8e964e77-4315-44b2-a34f-d0e2249e9a72","Type":"ContainerDied","Data":"77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1"} Oct 11 10:45:03.966445 master-1 kubenswrapper[4771]: I1011 10:45:03.966382 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-btlwb" event={"ID":"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1","Type":"ContainerStarted","Data":"b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809"} Oct 11 10:45:03.968896 master-1 kubenswrapper[4771]: I1011 10:45:03.968856 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b846b7bb4-xmv6l_a65b0165-5747-48c9-9179-86f19861dd68/console/0.log" Oct 11 10:45:03.969022 master-1 kubenswrapper[4771]: I1011 10:45:03.968926 4771 generic.go:334] "Generic (PLEG): container finished" podID="a65b0165-5747-48c9-9179-86f19861dd68" containerID="6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6" exitCode=2 Oct 11 10:45:03.969236 master-1 kubenswrapper[4771]: I1011 10:45:03.969072 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b846b7bb4-xmv6l" Oct 11 10:45:03.969445 master-1 kubenswrapper[4771]: I1011 10:45:03.969402 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b846b7bb4-xmv6l" event={"ID":"a65b0165-5747-48c9-9179-86f19861dd68","Type":"ContainerDied","Data":"6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6"} Oct 11 10:45:03.969548 master-1 kubenswrapper[4771]: I1011 10:45:03.969458 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b846b7bb4-xmv6l" event={"ID":"a65b0165-5747-48c9-9179-86f19861dd68","Type":"ContainerDied","Data":"2e652727629cf31e7de5014abdf61de5e97f13fd0cbfe170fa06452ef6ed0070"} Oct 11 10:45:03.969548 master-1 kubenswrapper[4771]: I1011 10:45:03.969490 4771 scope.go:117] "RemoveContainer" containerID="6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6" Oct 11 10:45:03.975509 master-1 kubenswrapper[4771]: I1011 10:45:03.975444 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-fn27x" event={"ID":"dd28168d-b375-4a82-8784-bc38fad4cc07","Type":"ContainerStarted","Data":"3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70"} Oct 11 10:45:04.000718 master-1 kubenswrapper[4771]: I1011 10:45:04.000646 4771 scope.go:117] "RemoveContainer" containerID="6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6" Oct 11 10:45:04.001344 master-1 kubenswrapper[4771]: E1011 10:45:04.001282 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6\": container with ID starting with 6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6 not found: ID does not exist" containerID="6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6" Oct 11 10:45:04.001470 master-1 kubenswrapper[4771]: I1011 10:45:04.001346 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6"} err="failed to get container status \"6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6\": rpc error: code = NotFound desc = could not find container \"6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6\": container with ID starting with 6930f37f5d40f6d2be4d6635242240c2b455d30927958a5c8bf12f960d07b1a6 not found: ID does not exist" Oct 11 10:45:04.023816 master-1 kubenswrapper[4771]: I1011 10:45:04.023716 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-marketplace-btlwb" podStartSLOduration=2.534020286 podStartE2EDuration="5.023688254s" podCreationTimestamp="2025-10-11 10:44:59 +0000 UTC" firstStartedPulling="2025-10-11 10:45:00.927553969 +0000 UTC m=+1132.901780410" lastFinishedPulling="2025-10-11 10:45:03.417221897 +0000 UTC m=+1135.391448378" observedRunningTime="2025-10-11 10:45:04.008464201 +0000 UTC m=+1135.982690722" watchObservedRunningTime="2025-10-11 10:45:04.023688254 +0000 UTC m=+1135.997914705" Oct 11 10:45:04.048393 master-1 kubenswrapper[4771]: I1011 10:45:04.046407 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-operators-fn27x" podStartSLOduration=2.500884562 podStartE2EDuration="5.04638057s" podCreationTimestamp="2025-10-11 10:44:59 +0000 UTC" firstStartedPulling="2025-10-11 10:45:00.930107912 +0000 UTC m=+1132.904334343" lastFinishedPulling="2025-10-11 10:45:03.47560391 +0000 UTC m=+1135.449830351" observedRunningTime="2025-10-11 10:45:04.041994535 +0000 UTC m=+1136.016220976" watchObservedRunningTime="2025-10-11 10:45:04.04638057 +0000 UTC m=+1136.020607021" Oct 11 10:45:04.068226 master-1 kubenswrapper[4771]: I1011 10:45:04.068139 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b846b7bb4-xmv6l"] Oct 11 10:45:04.071093 master-1 kubenswrapper[4771]: I1011 10:45:04.071037 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b846b7bb4-xmv6l"] Oct 11 10:45:04.445481 master-1 kubenswrapper[4771]: I1011 10:45:04.445399 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a65b0165-5747-48c9-9179-86f19861dd68" path="/var/lib/kubelet/pods/a65b0165-5747-48c9-9179-86f19861dd68/volumes" Oct 11 10:45:04.986952 master-1 kubenswrapper[4771]: I1011 10:45:04.986907 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-r8hdr" event={"ID":"8e964e77-4315-44b2-a34f-d0e2249e9a72","Type":"ContainerStarted","Data":"3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79"} Oct 11 10:45:05.999089 master-1 kubenswrapper[4771]: I1011 10:45:05.999004 4771 generic.go:334] "Generic (PLEG): container finished" podID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerID="3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79" exitCode=0 Oct 11 10:45:05.999089 master-1 kubenswrapper[4771]: I1011 10:45:05.999086 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-r8hdr" event={"ID":"8e964e77-4315-44b2-a34f-d0e2249e9a72","Type":"ContainerDied","Data":"3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79"} Oct 11 10:45:07.009898 master-1 kubenswrapper[4771]: I1011 10:45:07.009839 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-r8hdr" event={"ID":"8e964e77-4315-44b2-a34f-d0e2249e9a72","Type":"ContainerStarted","Data":"bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe"} Oct 11 10:45:07.122558 master-1 kubenswrapper[4771]: I1011 10:45:07.122416 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/community-operators-r8hdr" podStartSLOduration=3.571885533 podStartE2EDuration="6.122394094s" podCreationTimestamp="2025-10-11 10:45:01 +0000 UTC" firstStartedPulling="2025-10-11 10:45:03.963208122 +0000 UTC m=+1135.937434603" lastFinishedPulling="2025-10-11 10:45:06.513716693 +0000 UTC m=+1138.487943164" observedRunningTime="2025-10-11 10:45:07.11628978 +0000 UTC m=+1139.090516261" watchObservedRunningTime="2025-10-11 10:45:07.122394094 +0000 UTC m=+1139.096620545" Oct 11 10:45:09.755418 master-1 kubenswrapper[4771]: I1011 10:45:09.755203 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:45:09.755418 master-1 kubenswrapper[4771]: I1011 10:45:09.755302 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:45:09.823035 master-1 kubenswrapper[4771]: I1011 10:45:09.822965 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:45:09.950843 master-1 kubenswrapper[4771]: I1011 10:45:09.950749 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:45:09.950843 master-1 kubenswrapper[4771]: I1011 10:45:09.950825 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:45:10.016981 master-1 kubenswrapper[4771]: I1011 10:45:10.016813 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:45:10.095647 master-1 kubenswrapper[4771]: I1011 10:45:10.095567 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:45:10.101934 master-1 kubenswrapper[4771]: I1011 10:45:10.101851 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:45:12.184382 master-1 kubenswrapper[4771]: I1011 10:45:12.184297 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-fn27x"] Oct 11 10:45:12.185058 master-1 kubenswrapper[4771]: I1011 10:45:12.184638 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-operators-fn27x" podUID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerName="registry-server" containerID="cri-o://3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70" gracePeriod=2 Oct 11 10:45:12.334409 master-1 kubenswrapper[4771]: I1011 10:45:12.329913 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:12.334409 master-1 kubenswrapper[4771]: I1011 10:45:12.330212 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:12.400709 master-1 kubenswrapper[4771]: I1011 10:45:12.400606 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-btlwb"] Oct 11 10:45:12.401286 master-1 kubenswrapper[4771]: I1011 10:45:12.401223 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-marketplace-btlwb" podUID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerName="registry-server" containerID="cri-o://b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809" gracePeriod=2 Oct 11 10:45:12.413921 master-1 kubenswrapper[4771]: I1011 10:45:12.413797 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:12.609697 master-1 kubenswrapper[4771]: I1011 10:45:12.609610 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:45:12.766915 master-1 kubenswrapper[4771]: I1011 10:45:12.766786 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-catalog-content\") pod \"dd28168d-b375-4a82-8784-bc38fad4cc07\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " Oct 11 10:45:12.766915 master-1 kubenswrapper[4771]: I1011 10:45:12.766864 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-utilities\") pod \"dd28168d-b375-4a82-8784-bc38fad4cc07\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " Oct 11 10:45:12.766915 master-1 kubenswrapper[4771]: I1011 10:45:12.766895 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z8rx\" (UniqueName: \"kubernetes.io/projected/dd28168d-b375-4a82-8784-bc38fad4cc07-kube-api-access-4z8rx\") pod \"dd28168d-b375-4a82-8784-bc38fad4cc07\" (UID: \"dd28168d-b375-4a82-8784-bc38fad4cc07\") " Oct 11 10:45:12.768456 master-1 kubenswrapper[4771]: I1011 10:45:12.768382 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-utilities" (OuterVolumeSpecName: "utilities") pod "dd28168d-b375-4a82-8784-bc38fad4cc07" (UID: "dd28168d-b375-4a82-8784-bc38fad4cc07"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:45:12.773981 master-1 kubenswrapper[4771]: I1011 10:45:12.773908 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd28168d-b375-4a82-8784-bc38fad4cc07-kube-api-access-4z8rx" (OuterVolumeSpecName: "kube-api-access-4z8rx") pod "dd28168d-b375-4a82-8784-bc38fad4cc07" (UID: "dd28168d-b375-4a82-8784-bc38fad4cc07"). InnerVolumeSpecName "kube-api-access-4z8rx". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:45:12.812351 master-1 kubenswrapper[4771]: I1011 10:45:12.812287 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:45:12.868682 master-1 kubenswrapper[4771]: I1011 10:45:12.868631 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:12.868682 master-1 kubenswrapper[4771]: I1011 10:45:12.868663 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-4z8rx\" (UniqueName: \"kubernetes.io/projected/dd28168d-b375-4a82-8784-bc38fad4cc07-kube-api-access-4z8rx\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:12.969347 master-1 kubenswrapper[4771]: I1011 10:45:12.969256 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-catalog-content\") pod \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " Oct 11 10:45:12.969347 master-1 kubenswrapper[4771]: I1011 10:45:12.969345 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-utilities\") pod \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " Oct 11 10:45:12.969809 master-1 kubenswrapper[4771]: I1011 10:45:12.969496 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjs9q\" (UniqueName: \"kubernetes.io/projected/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-kube-api-access-jjs9q\") pod \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\" (UID: \"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1\") " Oct 11 10:45:12.971274 master-1 kubenswrapper[4771]: I1011 10:45:12.971206 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-utilities" (OuterVolumeSpecName: "utilities") pod "aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" (UID: "aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:45:12.972787 master-1 kubenswrapper[4771]: I1011 10:45:12.972698 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-kube-api-access-jjs9q" (OuterVolumeSpecName: "kube-api-access-jjs9q") pod "aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" (UID: "aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1"). InnerVolumeSpecName "kube-api-access-jjs9q". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:45:12.997015 master-1 kubenswrapper[4771]: I1011 10:45:12.996921 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" (UID: "aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:45:13.018265 master-1 kubenswrapper[4771]: I1011 10:45:13.018109 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "dd28168d-b375-4a82-8784-bc38fad4cc07" (UID: "dd28168d-b375-4a82-8784-bc38fad4cc07"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:45:13.058975 master-1 kubenswrapper[4771]: I1011 10:45:13.058878 4771 generic.go:334] "Generic (PLEG): container finished" podID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerID="3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70" exitCode=0 Oct 11 10:45:13.059265 master-1 kubenswrapper[4771]: I1011 10:45:13.059010 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-fn27x" event={"ID":"dd28168d-b375-4a82-8784-bc38fad4cc07","Type":"ContainerDied","Data":"3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70"} Oct 11 10:45:13.059265 master-1 kubenswrapper[4771]: I1011 10:45:13.059070 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-fn27x" event={"ID":"dd28168d-b375-4a82-8784-bc38fad4cc07","Type":"ContainerDied","Data":"6a5c5b9ff534c3108ab033d2a189e452b9f1cc8c2fa78f601306eadfc2f6563e"} Oct 11 10:45:13.059265 master-1 kubenswrapper[4771]: I1011 10:45:13.059106 4771 scope.go:117] "RemoveContainer" containerID="3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70" Oct 11 10:45:13.059265 master-1 kubenswrapper[4771]: I1011 10:45:13.059153 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-fn27x" Oct 11 10:45:13.064403 master-1 kubenswrapper[4771]: I1011 10:45:13.064313 4771 generic.go:334] "Generic (PLEG): container finished" podID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerID="b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809" exitCode=0 Oct 11 10:45:13.064533 master-1 kubenswrapper[4771]: I1011 10:45:13.064424 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-btlwb" event={"ID":"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1","Type":"ContainerDied","Data":"b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809"} Oct 11 10:45:13.064533 master-1 kubenswrapper[4771]: I1011 10:45:13.064486 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-btlwb" event={"ID":"aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1","Type":"ContainerDied","Data":"07f0178441f3d50314b9519115a4f3d7e321d0618d45e6c54bc9d3b3f3be9ea8"} Oct 11 10:45:13.064533 master-1 kubenswrapper[4771]: I1011 10:45:13.064510 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-btlwb" Oct 11 10:45:13.070968 master-1 kubenswrapper[4771]: I1011 10:45:13.070877 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:13.070968 master-1 kubenswrapper[4771]: I1011 10:45:13.070939 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:13.070968 master-1 kubenswrapper[4771]: I1011 10:45:13.070952 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-jjs9q\" (UniqueName: \"kubernetes.io/projected/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1-kube-api-access-jjs9q\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:13.070968 master-1 kubenswrapper[4771]: I1011 10:45:13.070963 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/dd28168d-b375-4a82-8784-bc38fad4cc07-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:13.080780 master-1 kubenswrapper[4771]: I1011 10:45:13.080713 4771 scope.go:117] "RemoveContainer" containerID="63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8" Oct 11 10:45:13.105780 master-1 kubenswrapper[4771]: I1011 10:45:13.105704 4771 scope.go:117] "RemoveContainer" containerID="09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132" Oct 11 10:45:13.128213 master-1 kubenswrapper[4771]: I1011 10:45:13.128139 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-fn27x"] Oct 11 10:45:13.131332 master-1 kubenswrapper[4771]: I1011 10:45:13.131266 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:13.131461 master-1 kubenswrapper[4771]: I1011 10:45:13.131370 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-operators-fn27x"] Oct 11 10:45:13.147890 master-1 kubenswrapper[4771]: I1011 10:45:13.147832 4771 scope.go:117] "RemoveContainer" containerID="3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70" Oct 11 10:45:13.150237 master-1 kubenswrapper[4771]: E1011 10:45:13.150171 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70\": container with ID starting with 3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70 not found: ID does not exist" containerID="3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70" Oct 11 10:45:13.150327 master-1 kubenswrapper[4771]: I1011 10:45:13.150249 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70"} err="failed to get container status \"3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70\": rpc error: code = NotFound desc = could not find container \"3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70\": container with ID starting with 3f320377dafbebce17a26632f3fc75e231a9d287fb6de8d7ce1701d394699b70 not found: ID does not exist" Oct 11 10:45:13.150327 master-1 kubenswrapper[4771]: I1011 10:45:13.150289 4771 scope.go:117] "RemoveContainer" containerID="63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8" Oct 11 10:45:13.150445 master-1 kubenswrapper[4771]: I1011 10:45:13.150335 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-btlwb"] Oct 11 10:45:13.151095 master-1 kubenswrapper[4771]: E1011 10:45:13.151048 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8\": container with ID starting with 63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8 not found: ID does not exist" containerID="63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8" Oct 11 10:45:13.151146 master-1 kubenswrapper[4771]: I1011 10:45:13.151101 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8"} err="failed to get container status \"63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8\": rpc error: code = NotFound desc = could not find container \"63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8\": container with ID starting with 63deab1a90696cd378fba377a0ccafb64b7d1abb648ebd498f4ae24255a846e8 not found: ID does not exist" Oct 11 10:45:13.151146 master-1 kubenswrapper[4771]: I1011 10:45:13.151134 4771 scope.go:117] "RemoveContainer" containerID="09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132" Oct 11 10:45:13.151604 master-1 kubenswrapper[4771]: E1011 10:45:13.151570 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132\": container with ID starting with 09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132 not found: ID does not exist" containerID="09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132" Oct 11 10:45:13.151604 master-1 kubenswrapper[4771]: I1011 10:45:13.151597 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132"} err="failed to get container status \"09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132\": rpc error: code = NotFound desc = could not find container \"09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132\": container with ID starting with 09ebc1d9dd212e1b780cdd4c208a385d6c32b3be45f5a7fd3316afb9285c5132 not found: ID does not exist" Oct 11 10:45:13.151706 master-1 kubenswrapper[4771]: I1011 10:45:13.151612 4771 scope.go:117] "RemoveContainer" containerID="b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809" Oct 11 10:45:13.155719 master-1 kubenswrapper[4771]: I1011 10:45:13.155645 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-marketplace-btlwb"] Oct 11 10:45:13.169062 master-1 kubenswrapper[4771]: I1011 10:45:13.168997 4771 scope.go:117] "RemoveContainer" containerID="b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d" Oct 11 10:45:13.188984 master-1 kubenswrapper[4771]: I1011 10:45:13.188741 4771 scope.go:117] "RemoveContainer" containerID="563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245" Oct 11 10:45:13.210240 master-1 kubenswrapper[4771]: I1011 10:45:13.210204 4771 scope.go:117] "RemoveContainer" containerID="b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809" Oct 11 10:45:13.211088 master-1 kubenswrapper[4771]: E1011 10:45:13.211024 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809\": container with ID starting with b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809 not found: ID does not exist" containerID="b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809" Oct 11 10:45:13.211147 master-1 kubenswrapper[4771]: I1011 10:45:13.211112 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809"} err="failed to get container status \"b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809\": rpc error: code = NotFound desc = could not find container \"b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809\": container with ID starting with b2ac95ce6b7361d31eadc268199ed3e400e0a7f86b2cc191c90b28cc26682809 not found: ID does not exist" Oct 11 10:45:13.211203 master-1 kubenswrapper[4771]: I1011 10:45:13.211175 4771 scope.go:117] "RemoveContainer" containerID="b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d" Oct 11 10:45:13.212316 master-1 kubenswrapper[4771]: E1011 10:45:13.212234 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d\": container with ID starting with b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d not found: ID does not exist" containerID="b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d" Oct 11 10:45:13.212446 master-1 kubenswrapper[4771]: I1011 10:45:13.212338 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d"} err="failed to get container status \"b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d\": rpc error: code = NotFound desc = could not find container \"b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d\": container with ID starting with b28d4d25a229e182db13cc45f68ab48fb16a28b5be04facbfd88980d7656d25d not found: ID does not exist" Oct 11 10:45:13.212506 master-1 kubenswrapper[4771]: I1011 10:45:13.212452 4771 scope.go:117] "RemoveContainer" containerID="563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245" Oct 11 10:45:13.213073 master-1 kubenswrapper[4771]: E1011 10:45:13.213021 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245\": container with ID starting with 563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245 not found: ID does not exist" containerID="563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245" Oct 11 10:45:13.213161 master-1 kubenswrapper[4771]: I1011 10:45:13.213070 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245"} err="failed to get container status \"563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245\": rpc error: code = NotFound desc = could not find container \"563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245\": container with ID starting with 563b4b269fc1aea55e95d3112d36370a783bc78f1b8ea1724512328627aa0245 not found: ID does not exist" Oct 11 10:45:13.455058 master-1 kubenswrapper[4771]: I1011 10:45:13.454875 4771 scope.go:117] "RemoveContainer" containerID="79e52bbf7393881dfbba04f7a9f71721266d98f1191a6c7be91f8bc0ce4e1139" Oct 11 10:45:13.474706 master-1 kubenswrapper[4771]: I1011 10:45:13.474648 4771 scope.go:117] "RemoveContainer" containerID="9e6a4086932c3b4c0590b1992411e46984c974a11450de3378bede5ca3045d02" Oct 11 10:45:13.496973 master-1 kubenswrapper[4771]: I1011 10:45:13.496891 4771 scope.go:117] "RemoveContainer" containerID="068b46162b2804f4e661290cc4e58111faa3ee64a5ff733b8a30de9f4b7d070e" Oct 11 10:45:13.524690 master-1 kubenswrapper[4771]: I1011 10:45:13.524586 4771 scope.go:117] "RemoveContainer" containerID="913e0c188082961ad93b5f6a07d9eda57e62160ccbff129947e77948c758035a" Oct 11 10:45:14.445040 master-1 kubenswrapper[4771]: I1011 10:45:14.444957 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" path="/var/lib/kubelet/pods/aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1/volumes" Oct 11 10:45:14.446508 master-1 kubenswrapper[4771]: I1011 10:45:14.446466 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd28168d-b375-4a82-8784-bc38fad4cc07" path="/var/lib/kubelet/pods/dd28168d-b375-4a82-8784-bc38fad4cc07/volumes" Oct 11 10:45:17.185090 master-1 kubenswrapper[4771]: I1011 10:45:17.184958 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-r8hdr"] Oct 11 10:45:17.187306 master-1 kubenswrapper[4771]: I1011 10:45:17.185495 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/community-operators-r8hdr" podUID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerName="registry-server" containerID="cri-o://bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe" gracePeriod=2 Oct 11 10:45:17.753046 master-1 kubenswrapper[4771]: I1011 10:45:17.752976 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:17.945414 master-1 kubenswrapper[4771]: I1011 10:45:17.945234 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plx55\" (UniqueName: \"kubernetes.io/projected/8e964e77-4315-44b2-a34f-d0e2249e9a72-kube-api-access-plx55\") pod \"8e964e77-4315-44b2-a34f-d0e2249e9a72\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " Oct 11 10:45:17.945414 master-1 kubenswrapper[4771]: I1011 10:45:17.945332 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-catalog-content\") pod \"8e964e77-4315-44b2-a34f-d0e2249e9a72\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " Oct 11 10:45:17.945655 master-1 kubenswrapper[4771]: I1011 10:45:17.945463 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-utilities\") pod \"8e964e77-4315-44b2-a34f-d0e2249e9a72\" (UID: \"8e964e77-4315-44b2-a34f-d0e2249e9a72\") " Oct 11 10:45:17.947410 master-1 kubenswrapper[4771]: I1011 10:45:17.947340 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-utilities" (OuterVolumeSpecName: "utilities") pod "8e964e77-4315-44b2-a34f-d0e2249e9a72" (UID: "8e964e77-4315-44b2-a34f-d0e2249e9a72"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:45:17.949842 master-1 kubenswrapper[4771]: I1011 10:45:17.949804 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e964e77-4315-44b2-a34f-d0e2249e9a72-kube-api-access-plx55" (OuterVolumeSpecName: "kube-api-access-plx55") pod "8e964e77-4315-44b2-a34f-d0e2249e9a72" (UID: "8e964e77-4315-44b2-a34f-d0e2249e9a72"). InnerVolumeSpecName "kube-api-access-plx55". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:45:18.017069 master-1 kubenswrapper[4771]: I1011 10:45:18.016988 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "8e964e77-4315-44b2-a34f-d0e2249e9a72" (UID: "8e964e77-4315-44b2-a34f-d0e2249e9a72"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:45:18.047329 master-1 kubenswrapper[4771]: I1011 10:45:18.047263 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-plx55\" (UniqueName: \"kubernetes.io/projected/8e964e77-4315-44b2-a34f-d0e2249e9a72-kube-api-access-plx55\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:18.047329 master-1 kubenswrapper[4771]: I1011 10:45:18.047321 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:18.047439 master-1 kubenswrapper[4771]: I1011 10:45:18.047341 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8e964e77-4315-44b2-a34f-d0e2249e9a72-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 10:45:18.112957 master-1 kubenswrapper[4771]: I1011 10:45:18.112859 4771 generic.go:334] "Generic (PLEG): container finished" podID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerID="bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe" exitCode=0 Oct 11 10:45:18.112957 master-1 kubenswrapper[4771]: I1011 10:45:18.112915 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-r8hdr" event={"ID":"8e964e77-4315-44b2-a34f-d0e2249e9a72","Type":"ContainerDied","Data":"bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe"} Oct 11 10:45:18.112957 master-1 kubenswrapper[4771]: I1011 10:45:18.112944 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-r8hdr" event={"ID":"8e964e77-4315-44b2-a34f-d0e2249e9a72","Type":"ContainerDied","Data":"7305b66b7aeba582a9f93dc93062bbbc5bb8eccd416f40fbbfbc9aebdb769b49"} Oct 11 10:45:18.112957 master-1 kubenswrapper[4771]: I1011 10:45:18.112964 4771 scope.go:117] "RemoveContainer" containerID="bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe" Oct 11 10:45:18.113594 master-1 kubenswrapper[4771]: I1011 10:45:18.113083 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-r8hdr" Oct 11 10:45:18.150700 master-1 kubenswrapper[4771]: I1011 10:45:18.150629 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-r8hdr"] Oct 11 10:45:18.154101 master-1 kubenswrapper[4771]: I1011 10:45:18.154044 4771 scope.go:117] "RemoveContainer" containerID="3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79" Oct 11 10:45:18.155056 master-1 kubenswrapper[4771]: I1011 10:45:18.155015 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/community-operators-r8hdr"] Oct 11 10:45:18.173944 master-1 kubenswrapper[4771]: I1011 10:45:18.173862 4771 scope.go:117] "RemoveContainer" containerID="77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1" Oct 11 10:45:18.202193 master-1 kubenswrapper[4771]: I1011 10:45:18.202148 4771 scope.go:117] "RemoveContainer" containerID="bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe" Oct 11 10:45:18.202642 master-1 kubenswrapper[4771]: E1011 10:45:18.202594 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe\": container with ID starting with bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe not found: ID does not exist" containerID="bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe" Oct 11 10:45:18.202704 master-1 kubenswrapper[4771]: I1011 10:45:18.202656 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe"} err="failed to get container status \"bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe\": rpc error: code = NotFound desc = could not find container \"bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe\": container with ID starting with bc926f519f674a84e8c91bddae5383960590b03479e2c4a4ef1e2497b3e6fbbe not found: ID does not exist" Oct 11 10:45:18.202747 master-1 kubenswrapper[4771]: I1011 10:45:18.202712 4771 scope.go:117] "RemoveContainer" containerID="3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79" Oct 11 10:45:18.203589 master-1 kubenswrapper[4771]: E1011 10:45:18.203518 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79\": container with ID starting with 3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79 not found: ID does not exist" containerID="3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79" Oct 11 10:45:18.203673 master-1 kubenswrapper[4771]: I1011 10:45:18.203608 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79"} err="failed to get container status \"3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79\": rpc error: code = NotFound desc = could not find container \"3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79\": container with ID starting with 3e6d711a5742f6f82fe5987f3befce4cbd02d09a7cb95428ec3ad9fce91b0d79 not found: ID does not exist" Oct 11 10:45:18.203715 master-1 kubenswrapper[4771]: I1011 10:45:18.203679 4771 scope.go:117] "RemoveContainer" containerID="77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1" Oct 11 10:45:18.204328 master-1 kubenswrapper[4771]: E1011 10:45:18.204283 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1\": container with ID starting with 77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1 not found: ID does not exist" containerID="77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1" Oct 11 10:45:18.204387 master-1 kubenswrapper[4771]: I1011 10:45:18.204331 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1"} err="failed to get container status \"77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1\": rpc error: code = NotFound desc = could not find container \"77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1\": container with ID starting with 77010f46cea21ca3cf2adec7d3a4536cce32a5cb226d001a2497b507dc532ac1 not found: ID does not exist" Oct 11 10:45:18.447231 master-1 kubenswrapper[4771]: I1011 10:45:18.447144 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e964e77-4315-44b2-a34f-d0e2249e9a72" path="/var/lib/kubelet/pods/8e964e77-4315-44b2-a34f-d0e2249e9a72/volumes" Oct 11 10:46:36.869371 master-1 kubenswrapper[4771]: I1011 10:46:36.869258 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-etcd/revision-pruner-10-master-1"] Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869514 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerName="registry-server" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869530 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerName="registry-server" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869542 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerName="registry-server" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869548 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerName="registry-server" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869561 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerName="extract-utilities" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869567 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerName="extract-utilities" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869576 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerName="extract-content" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869581 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerName="extract-content" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869589 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerName="extract-content" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869595 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerName="extract-content" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869605 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerName="extract-utilities" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869611 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerName="extract-utilities" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869622 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerName="extract-utilities" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869628 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerName="extract-utilities" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869637 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerName="registry-server" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869644 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerName="registry-server" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869651 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a65b0165-5747-48c9-9179-86f19861dd68" containerName="console" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869656 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65b0165-5747-48c9-9179-86f19861dd68" containerName="console" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: E1011 10:46:36.869664 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerName="extract-content" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869670 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerName="extract-content" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869749 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="aa7fddf6-d341-4992-bba8-9d5fa5b1e7a1" containerName="registry-server" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869761 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a65b0165-5747-48c9-9179-86f19861dd68" containerName="console" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869772 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="8e964e77-4315-44b2-a34f-d0e2249e9a72" containerName="registry-server" Oct 11 10:46:36.870051 master-1 kubenswrapper[4771]: I1011 10:46:36.869779 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="dd28168d-b375-4a82-8784-bc38fad4cc07" containerName="registry-server" Oct 11 10:46:36.871068 master-1 kubenswrapper[4771]: I1011 10:46:36.870224 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:36.873043 master-1 kubenswrapper[4771]: I1011 10:46:36.873017 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-etcd"/"installer-sa-dockercfg-xbqxb" Oct 11 10:46:36.884523 master-1 kubenswrapper[4771]: I1011 10:46:36.884460 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/revision-pruner-10-master-1"] Oct 11 10:46:36.977400 master-1 kubenswrapper[4771]: I1011 10:46:36.977307 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/a47a3143-b015-49c8-a15d-678e348b64e8-kube-api-access\") pod \"revision-pruner-10-master-1\" (UID: \"a47a3143-b015-49c8-a15d-678e348b64e8\") " pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:36.977400 master-1 kubenswrapper[4771]: I1011 10:46:36.977393 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a47a3143-b015-49c8-a15d-678e348b64e8-kubelet-dir\") pod \"revision-pruner-10-master-1\" (UID: \"a47a3143-b015-49c8-a15d-678e348b64e8\") " pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:37.079174 master-1 kubenswrapper[4771]: I1011 10:46:37.079054 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/a47a3143-b015-49c8-a15d-678e348b64e8-kube-api-access\") pod \"revision-pruner-10-master-1\" (UID: \"a47a3143-b015-49c8-a15d-678e348b64e8\") " pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:37.079528 master-1 kubenswrapper[4771]: I1011 10:46:37.079201 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a47a3143-b015-49c8-a15d-678e348b64e8-kubelet-dir\") pod \"revision-pruner-10-master-1\" (UID: \"a47a3143-b015-49c8-a15d-678e348b64e8\") " pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:37.079528 master-1 kubenswrapper[4771]: I1011 10:46:37.079423 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a47a3143-b015-49c8-a15d-678e348b64e8-kubelet-dir\") pod \"revision-pruner-10-master-1\" (UID: \"a47a3143-b015-49c8-a15d-678e348b64e8\") " pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:37.112467 master-1 kubenswrapper[4771]: I1011 10:46:37.112329 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/a47a3143-b015-49c8-a15d-678e348b64e8-kube-api-access\") pod \"revision-pruner-10-master-1\" (UID: \"a47a3143-b015-49c8-a15d-678e348b64e8\") " pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:37.221248 master-1 kubenswrapper[4771]: I1011 10:46:37.221073 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:37.686515 master-1 kubenswrapper[4771]: I1011 10:46:37.686459 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-etcd/revision-pruner-10-master-1"] Oct 11 10:46:37.693900 master-1 kubenswrapper[4771]: W1011 10:46:37.693857 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda47a3143_b015_49c8_a15d_678e348b64e8.slice/crio-ac5ce240b40d8370900f9879a98c097946af4b021ca8b155564363fc97fb42b3 WatchSource:0}: Error finding container ac5ce240b40d8370900f9879a98c097946af4b021ca8b155564363fc97fb42b3: Status 404 returned error can't find the container with id ac5ce240b40d8370900f9879a98c097946af4b021ca8b155564363fc97fb42b3 Oct 11 10:46:37.732323 master-1 kubenswrapper[4771]: I1011 10:46:37.732275 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/revision-pruner-10-master-1" event={"ID":"a47a3143-b015-49c8-a15d-678e348b64e8","Type":"ContainerStarted","Data":"ac5ce240b40d8370900f9879a98c097946af4b021ca8b155564363fc97fb42b3"} Oct 11 10:46:38.744193 master-1 kubenswrapper[4771]: I1011 10:46:38.744055 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/revision-pruner-10-master-1" event={"ID":"a47a3143-b015-49c8-a15d-678e348b64e8","Type":"ContainerStarted","Data":"9f30d0f3808a9d2757a05579682aa059fc93ed81f62653b81a10620484dbf824"} Oct 11 10:46:38.778975 master-1 kubenswrapper[4771]: I1011 10:46:38.778795 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-etcd/revision-pruner-10-master-1" podStartSLOduration=2.778721886 podStartE2EDuration="2.778721886s" podCreationTimestamp="2025-10-11 10:46:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:46:38.773888509 +0000 UTC m=+1230.748115010" watchObservedRunningTime="2025-10-11 10:46:38.778721886 +0000 UTC m=+1230.752948367" Oct 11 10:46:39.753424 master-1 kubenswrapper[4771]: I1011 10:46:39.753280 4771 generic.go:334] "Generic (PLEG): container finished" podID="a47a3143-b015-49c8-a15d-678e348b64e8" containerID="9f30d0f3808a9d2757a05579682aa059fc93ed81f62653b81a10620484dbf824" exitCode=0 Oct 11 10:46:39.753929 master-1 kubenswrapper[4771]: I1011 10:46:39.753414 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/revision-pruner-10-master-1" event={"ID":"a47a3143-b015-49c8-a15d-678e348b64e8","Type":"ContainerDied","Data":"9f30d0f3808a9d2757a05579682aa059fc93ed81f62653b81a10620484dbf824"} Oct 11 10:46:41.260208 master-1 kubenswrapper[4771]: I1011 10:46:41.260150 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:41.346206 master-1 kubenswrapper[4771]: I1011 10:46:41.345558 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/a47a3143-b015-49c8-a15d-678e348b64e8-kube-api-access\") pod \"a47a3143-b015-49c8-a15d-678e348b64e8\" (UID: \"a47a3143-b015-49c8-a15d-678e348b64e8\") " Oct 11 10:46:41.346436 master-1 kubenswrapper[4771]: I1011 10:46:41.346289 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a47a3143-b015-49c8-a15d-678e348b64e8-kubelet-dir\") pod \"a47a3143-b015-49c8-a15d-678e348b64e8\" (UID: \"a47a3143-b015-49c8-a15d-678e348b64e8\") " Oct 11 10:46:41.346520 master-1 kubenswrapper[4771]: I1011 10:46:41.346450 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/a47a3143-b015-49c8-a15d-678e348b64e8-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "a47a3143-b015-49c8-a15d-678e348b64e8" (UID: "a47a3143-b015-49c8-a15d-678e348b64e8"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:46:41.346903 master-1 kubenswrapper[4771]: I1011 10:46:41.346853 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a47a3143-b015-49c8-a15d-678e348b64e8-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:46:41.349888 master-1 kubenswrapper[4771]: I1011 10:46:41.349830 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47a3143-b015-49c8-a15d-678e348b64e8-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "a47a3143-b015-49c8-a15d-678e348b64e8" (UID: "a47a3143-b015-49c8-a15d-678e348b64e8"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:46:41.448263 master-1 kubenswrapper[4771]: I1011 10:46:41.448099 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/a47a3143-b015-49c8-a15d-678e348b64e8-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:46:41.775017 master-1 kubenswrapper[4771]: I1011 10:46:41.774871 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-etcd/revision-pruner-10-master-1" event={"ID":"a47a3143-b015-49c8-a15d-678e348b64e8","Type":"ContainerDied","Data":"ac5ce240b40d8370900f9879a98c097946af4b021ca8b155564363fc97fb42b3"} Oct 11 10:46:41.775017 master-1 kubenswrapper[4771]: I1011 10:46:41.774933 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac5ce240b40d8370900f9879a98c097946af4b021ca8b155564363fc97fb42b3" Oct 11 10:46:41.775017 master-1 kubenswrapper[4771]: I1011 10:46:41.774956 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-etcd/revision-pruner-10-master-1" Oct 11 10:46:42.899612 master-1 kubenswrapper[4771]: I1011 10:46:42.899520 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-etcd/installer-1-master-1"] Oct 11 10:46:42.905034 master-1 kubenswrapper[4771]: I1011 10:46:42.904956 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-etcd/installer-1-master-1"] Oct 11 10:46:44.444221 master-1 kubenswrapper[4771]: I1011 10:46:44.444148 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="826e1279-bc0d-426e-b6e0-5108268f340e" path="/var/lib/kubelet/pods/826e1279-bc0d-426e-b6e0-5108268f340e/volumes" Oct 11 10:47:07.551512 master-1 kubenswrapper[4771]: I1011 10:47:07.551440 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/installer-6-master-1"] Oct 11 10:47:07.552317 master-1 kubenswrapper[4771]: E1011 10:47:07.551682 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a47a3143-b015-49c8-a15d-678e348b64e8" containerName="pruner" Oct 11 10:47:07.552317 master-1 kubenswrapper[4771]: I1011 10:47:07.551695 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47a3143-b015-49c8-a15d-678e348b64e8" containerName="pruner" Oct 11 10:47:07.552317 master-1 kubenswrapper[4771]: I1011 10:47:07.551814 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a47a3143-b015-49c8-a15d-678e348b64e8" containerName="pruner" Oct 11 10:47:07.552436 master-1 kubenswrapper[4771]: I1011 10:47:07.552382 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.555268 master-1 kubenswrapper[4771]: I1011 10:47:07.555210 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-kube-apiserver"/"installer-sa-dockercfg-t28rg" Oct 11 10:47:07.568874 master-1 kubenswrapper[4771]: I1011 10:47:07.568798 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-6-master-1"] Oct 11 10:47:07.651386 master-1 kubenswrapper[4771]: I1011 10:47:07.651307 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-kubelet-dir\") pod \"installer-6-master-1\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.651736 master-1 kubenswrapper[4771]: I1011 10:47:07.651587 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/75f73eff-98a5-47a6-b15c-2338930444b9-kube-api-access\") pod \"installer-6-master-1\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.651736 master-1 kubenswrapper[4771]: I1011 10:47:07.651695 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-var-lock\") pod \"installer-6-master-1\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.752916 master-1 kubenswrapper[4771]: I1011 10:47:07.752803 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/75f73eff-98a5-47a6-b15c-2338930444b9-kube-api-access\") pod \"installer-6-master-1\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.752916 master-1 kubenswrapper[4771]: I1011 10:47:07.752874 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-var-lock\") pod \"installer-6-master-1\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.752916 master-1 kubenswrapper[4771]: I1011 10:47:07.752942 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-kubelet-dir\") pod \"installer-6-master-1\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.754205 master-1 kubenswrapper[4771]: I1011 10:47:07.753037 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-kubelet-dir\") pod \"installer-6-master-1\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.754205 master-1 kubenswrapper[4771]: I1011 10:47:07.753083 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-var-lock\") pod \"installer-6-master-1\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.779896 master-1 kubenswrapper[4771]: I1011 10:47:07.779843 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/75f73eff-98a5-47a6-b15c-2338930444b9-kube-api-access\") pod \"installer-6-master-1\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:07.870188 master-1 kubenswrapper[4771]: I1011 10:47:07.870039 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:08.356153 master-1 kubenswrapper[4771]: I1011 10:47:08.356065 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/installer-6-master-1"] Oct 11 10:47:08.367185 master-1 kubenswrapper[4771]: W1011 10:47:08.367119 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod75f73eff_98a5_47a6_b15c_2338930444b9.slice/crio-ddde6b9d12870f560724e081c5814d541ef4de69b4025dd3e9fe28d6514d1fa5 WatchSource:0}: Error finding container ddde6b9d12870f560724e081c5814d541ef4de69b4025dd3e9fe28d6514d1fa5: Status 404 returned error can't find the container with id ddde6b9d12870f560724e081c5814d541ef4de69b4025dd3e9fe28d6514d1fa5 Oct 11 10:47:08.977311 master-1 kubenswrapper[4771]: I1011 10:47:08.977113 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-6-master-1" event={"ID":"75f73eff-98a5-47a6-b15c-2338930444b9","Type":"ContainerStarted","Data":"2a911b9063702410de3ab33cdd411750a6c1735b8b89a04533d1fce1ca985ed4"} Oct 11 10:47:08.978122 master-1 kubenswrapper[4771]: I1011 10:47:08.978058 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-6-master-1" event={"ID":"75f73eff-98a5-47a6-b15c-2338930444b9","Type":"ContainerStarted","Data":"ddde6b9d12870f560724e081c5814d541ef4de69b4025dd3e9fe28d6514d1fa5"} Oct 11 10:47:09.004653 master-1 kubenswrapper[4771]: I1011 10:47:09.004535 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/installer-6-master-1" podStartSLOduration=2.004514185 podStartE2EDuration="2.004514185s" podCreationTimestamp="2025-10-11 10:47:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:47:09.003522716 +0000 UTC m=+1260.977749167" watchObservedRunningTime="2025-10-11 10:47:09.004514185 +0000 UTC m=+1260.978740636" Oct 11 10:47:13.652485 master-1 kubenswrapper[4771]: I1011 10:47:13.652114 4771 scope.go:117] "RemoveContainer" containerID="9a616ae6ac6ffcbc27ae54a54aec1c65046926d3773ee73ab8bfdedb75371f06" Oct 11 10:47:46.753440 master-1 kubenswrapper[4771]: I1011 10:47:46.753336 4771 kubelet.go:2431] "SyncLoop REMOVE" source="file" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:47:46.754349 master-1 kubenswrapper[4771]: I1011 10:47:46.753732 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver" containerID="cri-o://d035b13d9431b1216e273c4ac7fb5eb87624d8740b70d29326082336302e3b46" gracePeriod=135 Oct 11 10:47:46.754349 master-1 kubenswrapper[4771]: I1011 10:47:46.753868 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-cert-syncer" containerID="cri-o://7e5a3711f36461fe4ced62a6738267cdf151c6f22d750936a4256bced2e89c2a" gracePeriod=135 Oct 11 10:47:46.754349 master-1 kubenswrapper[4771]: I1011 10:47:46.753821 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-check-endpoints" containerID="cri-o://a15e7539d2a0c42e8c6c8995bf98ff26ca0f322daf83394df48b4f13fc42d10b" gracePeriod=135 Oct 11 10:47:46.754349 master-1 kubenswrapper[4771]: I1011 10:47:46.753929 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-insecure-readyz" containerID="cri-o://452189c1a156cff2357db3338f99f86d41c76ed0f97b4459672ad6a8fe0dc5c7" gracePeriod=135 Oct 11 10:47:46.754349 master-1 kubenswrapper[4771]: I1011 10:47:46.753866 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-cert-regeneration-controller" containerID="cri-o://55ecf6fefa862d92619ce534057ad20c836371d13f4c0d70468214b0bd6e3db4" gracePeriod=135 Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: I1011 10:47:46.754807 4771 kubelet.go:2421] "SyncLoop ADD" source="file" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: E1011 10:47:46.755117 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: I1011 10:47:46.755138 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: E1011 10:47:46.755162 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-cert-regeneration-controller" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: I1011 10:47:46.755175 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-cert-regeneration-controller" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: E1011 10:47:46.755189 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-insecure-readyz" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: I1011 10:47:46.755201 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-insecure-readyz" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: E1011 10:47:46.755222 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-cert-syncer" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: I1011 10:47:46.755234 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-cert-syncer" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: E1011 10:47:46.755268 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="42d61efaa0f96869cf2939026aad6022" containerName="setup" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: I1011 10:47:46.755280 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d61efaa0f96869cf2939026aad6022" containerName="setup" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: E1011 10:47:46.755294 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-check-endpoints" Oct 11 10:47:46.755473 master-1 kubenswrapper[4771]: I1011 10:47:46.755308 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-check-endpoints" Oct 11 10:47:46.756001 master-1 kubenswrapper[4771]: I1011 10:47:46.755517 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-check-endpoints" Oct 11 10:47:46.756001 master-1 kubenswrapper[4771]: I1011 10:47:46.755544 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-cert-syncer" Oct 11 10:47:46.756001 master-1 kubenswrapper[4771]: I1011 10:47:46.755559 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-insecure-readyz" Oct 11 10:47:46.756001 master-1 kubenswrapper[4771]: I1011 10:47:46.755582 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver" Oct 11 10:47:46.756001 master-1 kubenswrapper[4771]: I1011 10:47:46.755598 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="42d61efaa0f96869cf2939026aad6022" containerName="kube-apiserver-cert-regeneration-controller" Oct 11 10:47:46.877984 master-1 kubenswrapper[4771]: I1011 10:47:46.877845 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/23141951a25391899fad7b9f2d5b6739-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"23141951a25391899fad7b9f2d5b6739\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:47:46.877984 master-1 kubenswrapper[4771]: I1011 10:47:46.877926 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/23141951a25391899fad7b9f2d5b6739-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"23141951a25391899fad7b9f2d5b6739\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:47:46.877984 master-1 kubenswrapper[4771]: I1011 10:47:46.877979 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/23141951a25391899fad7b9f2d5b6739-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"23141951a25391899fad7b9f2d5b6739\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:47:46.978699 master-1 kubenswrapper[4771]: I1011 10:47:46.978655 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/23141951a25391899fad7b9f2d5b6739-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"23141951a25391899fad7b9f2d5b6739\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:47:46.978783 master-1 kubenswrapper[4771]: I1011 10:47:46.978702 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/23141951a25391899fad7b9f2d5b6739-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"23141951a25391899fad7b9f2d5b6739\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:47:46.978783 master-1 kubenswrapper[4771]: I1011 10:47:46.978732 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/23141951a25391899fad7b9f2d5b6739-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"23141951a25391899fad7b9f2d5b6739\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:47:46.978913 master-1 kubenswrapper[4771]: I1011 10:47:46.978809 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/23141951a25391899fad7b9f2d5b6739-audit-dir\") pod \"kube-apiserver-master-1\" (UID: \"23141951a25391899fad7b9f2d5b6739\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:47:46.978913 master-1 kubenswrapper[4771]: I1011 10:47:46.978811 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/23141951a25391899fad7b9f2d5b6739-cert-dir\") pod \"kube-apiserver-master-1\" (UID: \"23141951a25391899fad7b9f2d5b6739\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:47:46.978913 master-1 kubenswrapper[4771]: I1011 10:47:46.978887 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/23141951a25391899fad7b9f2d5b6739-resource-dir\") pod \"kube-apiserver-master-1\" (UID: \"23141951a25391899fad7b9f2d5b6739\") " pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:47:47.262629 master-1 kubenswrapper[4771]: I1011 10:47:47.262531 4771 generic.go:334] "Generic (PLEG): container finished" podID="75f73eff-98a5-47a6-b15c-2338930444b9" containerID="2a911b9063702410de3ab33cdd411750a6c1735b8b89a04533d1fce1ca985ed4" exitCode=0 Oct 11 10:47:47.262629 master-1 kubenswrapper[4771]: I1011 10:47:47.262596 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-6-master-1" event={"ID":"75f73eff-98a5-47a6-b15c-2338930444b9","Type":"ContainerDied","Data":"2a911b9063702410de3ab33cdd411750a6c1735b8b89a04533d1fce1ca985ed4"} Oct 11 10:47:47.266892 master-1 kubenswrapper[4771]: I1011 10:47:47.266833 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_42d61efaa0f96869cf2939026aad6022/kube-apiserver-cert-syncer/0.log" Oct 11 10:47:47.268082 master-1 kubenswrapper[4771]: I1011 10:47:47.268023 4771 generic.go:334] "Generic (PLEG): container finished" podID="42d61efaa0f96869cf2939026aad6022" containerID="a15e7539d2a0c42e8c6c8995bf98ff26ca0f322daf83394df48b4f13fc42d10b" exitCode=0 Oct 11 10:47:47.268082 master-1 kubenswrapper[4771]: I1011 10:47:47.268071 4771 generic.go:334] "Generic (PLEG): container finished" podID="42d61efaa0f96869cf2939026aad6022" containerID="452189c1a156cff2357db3338f99f86d41c76ed0f97b4459672ad6a8fe0dc5c7" exitCode=0 Oct 11 10:47:47.268323 master-1 kubenswrapper[4771]: I1011 10:47:47.268087 4771 generic.go:334] "Generic (PLEG): container finished" podID="42d61efaa0f96869cf2939026aad6022" containerID="55ecf6fefa862d92619ce534057ad20c836371d13f4c0d70468214b0bd6e3db4" exitCode=0 Oct 11 10:47:47.268323 master-1 kubenswrapper[4771]: I1011 10:47:47.268102 4771 generic.go:334] "Generic (PLEG): container finished" podID="42d61efaa0f96869cf2939026aad6022" containerID="7e5a3711f36461fe4ced62a6738267cdf151c6f22d750936a4256bced2e89c2a" exitCode=2 Oct 11 10:47:47.294504 master-1 kubenswrapper[4771]: I1011 10:47:47.294198 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="42d61efaa0f96869cf2939026aad6022" podUID="23141951a25391899fad7b9f2d5b6739" Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: I1011 10:47:48.247036 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:47:48.247110 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:47:48.250178 master-1 kubenswrapper[4771]: I1011 10:47:48.247123 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:47:48.675727 master-1 kubenswrapper[4771]: I1011 10:47:48.675693 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:48.806870 master-1 kubenswrapper[4771]: I1011 10:47:48.806772 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/75f73eff-98a5-47a6-b15c-2338930444b9-kube-api-access\") pod \"75f73eff-98a5-47a6-b15c-2338930444b9\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " Oct 11 10:47:48.806870 master-1 kubenswrapper[4771]: I1011 10:47:48.806867 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-var-lock\") pod \"75f73eff-98a5-47a6-b15c-2338930444b9\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " Oct 11 10:47:48.807169 master-1 kubenswrapper[4771]: I1011 10:47:48.806982 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-kubelet-dir\") pod \"75f73eff-98a5-47a6-b15c-2338930444b9\" (UID: \"75f73eff-98a5-47a6-b15c-2338930444b9\") " Oct 11 10:47:48.807231 master-1 kubenswrapper[4771]: I1011 10:47:48.807147 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-var-lock" (OuterVolumeSpecName: "var-lock") pod "75f73eff-98a5-47a6-b15c-2338930444b9" (UID: "75f73eff-98a5-47a6-b15c-2338930444b9"). InnerVolumeSpecName "var-lock". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:47:48.807309 master-1 kubenswrapper[4771]: I1011 10:47:48.807209 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "75f73eff-98a5-47a6-b15c-2338930444b9" (UID: "75f73eff-98a5-47a6-b15c-2338930444b9"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:47:48.807679 master-1 kubenswrapper[4771]: I1011 10:47:48.807634 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:47:48.807679 master-1 kubenswrapper[4771]: I1011 10:47:48.807671 4771 reconciler_common.go:293] "Volume detached for volume \"var-lock\" (UniqueName: \"kubernetes.io/host-path/75f73eff-98a5-47a6-b15c-2338930444b9-var-lock\") on node \"master-1\" DevicePath \"\"" Oct 11 10:47:48.812741 master-1 kubenswrapper[4771]: I1011 10:47:48.812685 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f73eff-98a5-47a6-b15c-2338930444b9-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "75f73eff-98a5-47a6-b15c-2338930444b9" (UID: "75f73eff-98a5-47a6-b15c-2338930444b9"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:47:48.909412 master-1 kubenswrapper[4771]: I1011 10:47:48.909245 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/75f73eff-98a5-47a6-b15c-2338930444b9-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:47:49.286189 master-1 kubenswrapper[4771]: I1011 10:47:49.286045 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/installer-6-master-1" event={"ID":"75f73eff-98a5-47a6-b15c-2338930444b9","Type":"ContainerDied","Data":"ddde6b9d12870f560724e081c5814d541ef4de69b4025dd3e9fe28d6514d1fa5"} Oct 11 10:47:49.287466 master-1 kubenswrapper[4771]: I1011 10:47:49.287424 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddde6b9d12870f560724e081c5814d541ef4de69b4025dd3e9fe28d6514d1fa5" Oct 11 10:47:49.287665 master-1 kubenswrapper[4771]: I1011 10:47:49.286209 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/installer-6-master-1" Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: I1011 10:47:53.246229 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:47:53.246322 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:47:53.249810 master-1 kubenswrapper[4771]: I1011 10:47:53.246331 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: I1011 10:47:58.244545 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:47:58.244614 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:47:58.246871 master-1 kubenswrapper[4771]: I1011 10:47:58.246816 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:47:58.247095 master-1 kubenswrapper[4771]: I1011 10:47:58.247077 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: I1011 10:47:58.251975 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:47:58.252000 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:47:58.253485 master-1 kubenswrapper[4771]: I1011 10:47:58.253461 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: I1011 10:48:03.252140 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:03.252211 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:03.254558 master-1 kubenswrapper[4771]: I1011 10:48:03.252251 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: I1011 10:48:08.243271 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:08.243381 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:08.245320 master-1 kubenswrapper[4771]: I1011 10:48:08.243438 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: I1011 10:48:13.246020 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:13.246122 master-1 kubenswrapper[4771]: I1011 10:48:13.246090 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: I1011 10:48:18.243401 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:18.243488 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:18.246229 master-1 kubenswrapper[4771]: I1011 10:48:18.243509 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: I1011 10:48:23.245094 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:23.245729 master-1 kubenswrapper[4771]: I1011 10:48:23.245194 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: I1011 10:48:28.243264 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:28.243345 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:28.245056 master-1 kubenswrapper[4771]: I1011 10:48:28.243348 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: I1011 10:48:33.242270 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:33.242482 master-1 kubenswrapper[4771]: I1011 10:48:33.242406 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: I1011 10:48:38.243261 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:38.243387 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:38.247474 master-1 kubenswrapper[4771]: I1011 10:48:38.243347 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: I1011 10:48:43.245916 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:43.246011 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:43.248172 master-1 kubenswrapper[4771]: I1011 10:48:43.246017 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: I1011 10:48:48.245162 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:48.245218 master-1 kubenswrapper[4771]: I1011 10:48:48.245233 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: I1011 10:48:53.245399 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]log ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]api-openshift-apiserver-available ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]api-openshift-oauth-apiserver-available ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]informer-sync ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-api-request-count-filter ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-startkubeinformers ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-openshift-apiserver-reachable ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/openshift.io-oauth-apiserver-reachable ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiserver-admission-initializer ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/quota.openshift.io-clusterquotamapping ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/generic-apiserver-start-informers ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-consumer ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-filter ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/storage-object-count-tracker-hook ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-informers ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-apiextensions-controllers ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/crd-informer-synced ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-system-namespaces-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-cluster-authentication-info-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-legacy-token-tracking-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-service-ip-repair-controllers ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/rbac/bootstrap-roles ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/scheduling/bootstrap-system-priority-classes ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/priority-and-fairness-config-producer ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/bootstrap-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/aggregator-reload-proxy-client-cert ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/start-kube-aggregator-informers ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-local-available-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-status-remote-available-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-registration-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-wait-for-first-sync ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-discovery-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/kube-apiserver-autoregistration ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]autoregister-completion ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapi-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [+]poststarthook/apiservice-openapiv3-controller ok Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: [-]shutdown failed: reason withheld Oct 11 10:48:53.245518 master-1 kubenswrapper[4771]: readyz check failed Oct 11 10:48:53.248791 master-1 kubenswrapper[4771]: I1011 10:48:53.245515 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 11 10:48:54.124383 master-1 kubenswrapper[4771]: I1011 10:48:54.124247 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["metallb-system/frr-k8s-lvzhx"] Oct 11 10:48:54.124951 master-1 kubenswrapper[4771]: E1011 10:48:54.124614 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="75f73eff-98a5-47a6-b15c-2338930444b9" containerName="installer" Oct 11 10:48:54.124951 master-1 kubenswrapper[4771]: I1011 10:48:54.124636 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f73eff-98a5-47a6-b15c-2338930444b9" containerName="installer" Oct 11 10:48:54.124951 master-1 kubenswrapper[4771]: I1011 10:48:54.124775 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="75f73eff-98a5-47a6-b15c-2338930444b9" containerName="installer" Oct 11 10:48:54.127261 master-1 kubenswrapper[4771]: I1011 10:48:54.127208 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.130200 master-1 kubenswrapper[4771]: I1011 10:48:54.130141 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"metallb-system"/"frr-startup" Oct 11 10:48:54.131655 master-1 kubenswrapper[4771]: I1011 10:48:54.131384 4771 reflector.go:368] Caches populated for *v1.Secret from object-"metallb-system"/"frr-k8s-certs-secret" Oct 11 10:48:54.131655 master-1 kubenswrapper[4771]: I1011 10:48:54.131490 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"metallb-system"/"kube-root-ca.crt" Oct 11 10:48:54.132675 master-1 kubenswrapper[4771]: I1011 10:48:54.131828 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"metallb-system"/"openshift-service-ca.crt" Oct 11 10:48:54.313177 master-1 kubenswrapper[4771]: I1011 10:48:54.313096 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"frr-startup\" (UniqueName: \"kubernetes.io/configmap/f1e63653-b356-4bf6-b91a-6d386b1f3c33-frr-startup\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.313177 master-1 kubenswrapper[4771]: I1011 10:48:54.313160 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"reloader\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-reloader\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.313177 master-1 kubenswrapper[4771]: I1011 10:48:54.313190 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6tk\" (UniqueName: \"kubernetes.io/projected/f1e63653-b356-4bf6-b91a-6d386b1f3c33-kube-api-access-8x6tk\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.313871 master-1 kubenswrapper[4771]: I1011 10:48:54.313231 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"frr-sockets\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-frr-sockets\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.313871 master-1 kubenswrapper[4771]: I1011 10:48:54.313258 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e63653-b356-4bf6-b91a-6d386b1f3c33-metrics-certs\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.313871 master-1 kubenswrapper[4771]: I1011 10:48:54.313282 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"frr-conf\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-frr-conf\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.313871 master-1 kubenswrapper[4771]: I1011 10:48:54.313488 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-metrics\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.415887 master-1 kubenswrapper[4771]: I1011 10:48:54.415696 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"frr-startup\" (UniqueName: \"kubernetes.io/configmap/f1e63653-b356-4bf6-b91a-6d386b1f3c33-frr-startup\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.415887 master-1 kubenswrapper[4771]: I1011 10:48:54.415789 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"reloader\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-reloader\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.415887 master-1 kubenswrapper[4771]: I1011 10:48:54.415825 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6tk\" (UniqueName: \"kubernetes.io/projected/f1e63653-b356-4bf6-b91a-6d386b1f3c33-kube-api-access-8x6tk\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.415887 master-1 kubenswrapper[4771]: I1011 10:48:54.415859 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"frr-sockets\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-frr-sockets\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.415887 master-1 kubenswrapper[4771]: I1011 10:48:54.415885 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e63653-b356-4bf6-b91a-6d386b1f3c33-metrics-certs\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.415887 master-1 kubenswrapper[4771]: I1011 10:48:54.415910 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-metrics\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.416382 master-1 kubenswrapper[4771]: I1011 10:48:54.415932 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"frr-conf\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-frr-conf\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.416674 master-1 kubenswrapper[4771]: I1011 10:48:54.416608 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"frr-sockets\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-frr-sockets\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.416755 master-1 kubenswrapper[4771]: I1011 10:48:54.416717 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"reloader\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-reloader\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.416936 master-1 kubenswrapper[4771]: I1011 10:48:54.416894 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"frr-conf\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-frr-conf\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.417207 master-1 kubenswrapper[4771]: I1011 10:48:54.417173 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"frr-startup\" (UniqueName: \"kubernetes.io/configmap/f1e63653-b356-4bf6-b91a-6d386b1f3c33-frr-startup\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.417261 master-1 kubenswrapper[4771]: I1011 10:48:54.417159 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/empty-dir/f1e63653-b356-4bf6-b91a-6d386b1f3c33-metrics\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.422162 master-1 kubenswrapper[4771]: I1011 10:48:54.422082 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e63653-b356-4bf6-b91a-6d386b1f3c33-metrics-certs\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.519452 master-1 kubenswrapper[4771]: I1011 10:48:54.518853 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6tk\" (UniqueName: \"kubernetes.io/projected/f1e63653-b356-4bf6-b91a-6d386b1f3c33-kube-api-access-8x6tk\") pod \"frr-k8s-lvzhx\" (UID: \"f1e63653-b356-4bf6-b91a-6d386b1f3c33\") " pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.757663 master-1 kubenswrapper[4771]: I1011 10:48:54.757344 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:48:54.927006 master-1 kubenswrapper[4771]: I1011 10:48:54.925186 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["metallb-system/speaker-524kt"] Oct 11 10:48:54.927509 master-1 kubenswrapper[4771]: I1011 10:48:54.927475 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="metallb-system/speaker-524kt" Oct 11 10:48:54.930934 master-1 kubenswrapper[4771]: I1011 10:48:54.930895 4771 reflector.go:368] Caches populated for *v1.Secret from object-"metallb-system"/"metallb-memberlist" Oct 11 10:48:54.930995 master-1 kubenswrapper[4771]: I1011 10:48:54.930968 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"metallb-system"/"metallb-excludel2" Oct 11 10:48:54.931307 master-1 kubenswrapper[4771]: I1011 10:48:54.931271 4771 reflector.go:368] Caches populated for *v1.Secret from object-"metallb-system"/"speaker-certs-secret" Oct 11 10:48:55.126324 master-1 kubenswrapper[4771]: I1011 10:48:55.126208 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metallb-excludel2\" (UniqueName: \"kubernetes.io/configmap/72f01dc6-72cd-4eb0-8039-57150e0758bf-metallb-excludel2\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.126324 master-1 kubenswrapper[4771]: I1011 10:48:55.126329 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-metrics-certs\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.127206 master-1 kubenswrapper[4771]: I1011 10:48:55.126495 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"memberlist\" (UniqueName: \"kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-memberlist\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.127206 master-1 kubenswrapper[4771]: I1011 10:48:55.126561 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmqn9\" (UniqueName: \"kubernetes.io/projected/72f01dc6-72cd-4eb0-8039-57150e0758bf-kube-api-access-nmqn9\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.228326 master-1 kubenswrapper[4771]: I1011 10:48:55.228180 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metallb-excludel2\" (UniqueName: \"kubernetes.io/configmap/72f01dc6-72cd-4eb0-8039-57150e0758bf-metallb-excludel2\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.228326 master-1 kubenswrapper[4771]: I1011 10:48:55.228297 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-metrics-certs\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.228862 master-1 kubenswrapper[4771]: I1011 10:48:55.228351 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"memberlist\" (UniqueName: \"kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-memberlist\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.228862 master-1 kubenswrapper[4771]: I1011 10:48:55.228424 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqn9\" (UniqueName: \"kubernetes.io/projected/72f01dc6-72cd-4eb0-8039-57150e0758bf-kube-api-access-nmqn9\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.228862 master-1 kubenswrapper[4771]: E1011 10:48:55.228655 4771 secret.go:189] Couldn't get secret metallb-system/metallb-memberlist: secret "metallb-memberlist" not found Oct 11 10:48:55.228862 master-1 kubenswrapper[4771]: E1011 10:48:55.228807 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-memberlist podName:72f01dc6-72cd-4eb0-8039-57150e0758bf nodeName:}" failed. No retries permitted until 2025-10-11 10:48:55.728769685 +0000 UTC m=+1367.702996166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "memberlist" (UniqueName: "kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-memberlist") pod "speaker-524kt" (UID: "72f01dc6-72cd-4eb0-8039-57150e0758bf") : secret "metallb-memberlist" not found Oct 11 10:48:55.230418 master-1 kubenswrapper[4771]: I1011 10:48:55.230328 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metallb-excludel2\" (UniqueName: \"kubernetes.io/configmap/72f01dc6-72cd-4eb0-8039-57150e0758bf-metallb-excludel2\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.233892 master-1 kubenswrapper[4771]: I1011 10:48:55.233795 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-metrics-certs\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.264014 master-1 kubenswrapper[4771]: I1011 10:48:55.263863 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmqn9\" (UniqueName: \"kubernetes.io/projected/72f01dc6-72cd-4eb0-8039-57150e0758bf-kube-api-access-nmqn9\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.738088 master-1 kubenswrapper[4771]: I1011 10:48:55.738016 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"memberlist\" (UniqueName: \"kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-memberlist\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:55.738588 master-1 kubenswrapper[4771]: E1011 10:48:55.738298 4771 secret.go:189] Couldn't get secret metallb-system/metallb-memberlist: secret "metallb-memberlist" not found Oct 11 10:48:55.738588 master-1 kubenswrapper[4771]: E1011 10:48:55.738406 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-memberlist podName:72f01dc6-72cd-4eb0-8039-57150e0758bf nodeName:}" failed. No retries permitted until 2025-10-11 10:48:56.738381462 +0000 UTC m=+1368.712607943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "memberlist" (UniqueName: "kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-memberlist") pod "speaker-524kt" (UID: "72f01dc6-72cd-4eb0-8039-57150e0758bf") : secret "metallb-memberlist" not found Oct 11 10:48:55.749338 master-1 kubenswrapper[4771]: I1011 10:48:55.749287 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerStarted","Data":"d6423b0468f14f6975932098395821066a281fec17a5714650148996d09e9caa"} Oct 11 10:48:56.757302 master-1 kubenswrapper[4771]: I1011 10:48:56.757243 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"memberlist\" (UniqueName: \"kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-memberlist\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:56.763105 master-1 kubenswrapper[4771]: I1011 10:48:56.762930 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"memberlist\" (UniqueName: \"kubernetes.io/secret/72f01dc6-72cd-4eb0-8039-57150e0758bf-memberlist\") pod \"speaker-524kt\" (UID: \"72f01dc6-72cd-4eb0-8039-57150e0758bf\") " pod="metallb-system/speaker-524kt" Oct 11 10:48:56.905250 master-1 kubenswrapper[4771]: I1011 10:48:56.905172 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-nmstate/nmstate-handler-djsq6"] Oct 11 10:48:56.906102 master-1 kubenswrapper[4771]: I1011 10:48:56.906077 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:56.919689 master-1 kubenswrapper[4771]: I1011 10:48:56.919632 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-nmstate"/"kube-root-ca.crt" Oct 11 10:48:56.921836 master-1 kubenswrapper[4771]: I1011 10:48:56.919913 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-nmstate"/"openshift-service-ca.crt" Oct 11 10:48:56.960612 master-1 kubenswrapper[4771]: I1011 10:48:56.960504 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl5pt\" (UniqueName: \"kubernetes.io/projected/3e92427e-68a9-4496-9578-a0386bd5f5b3-kube-api-access-vl5pt\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:56.961208 master-1 kubenswrapper[4771]: I1011 10:48:56.960645 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nmstate-lock\" (UniqueName: \"kubernetes.io/host-path/3e92427e-68a9-4496-9578-a0386bd5f5b3-nmstate-lock\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:56.961208 master-1 kubenswrapper[4771]: I1011 10:48:56.960687 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovs-socket\" (UniqueName: \"kubernetes.io/host-path/3e92427e-68a9-4496-9578-a0386bd5f5b3-ovs-socket\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:56.961208 master-1 kubenswrapper[4771]: I1011 10:48:56.960718 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus-socket\" (UniqueName: \"kubernetes.io/host-path/3e92427e-68a9-4496-9578-a0386bd5f5b3-dbus-socket\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.047818 master-1 kubenswrapper[4771]: I1011 10:48:57.047672 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="metallb-system/speaker-524kt" Oct 11 10:48:57.064281 master-1 kubenswrapper[4771]: I1011 10:48:57.064225 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nmstate-lock\" (UniqueName: \"kubernetes.io/host-path/3e92427e-68a9-4496-9578-a0386bd5f5b3-nmstate-lock\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.064281 master-1 kubenswrapper[4771]: I1011 10:48:57.064285 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovs-socket\" (UniqueName: \"kubernetes.io/host-path/3e92427e-68a9-4496-9578-a0386bd5f5b3-ovs-socket\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.064622 master-1 kubenswrapper[4771]: I1011 10:48:57.064314 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dbus-socket\" (UniqueName: \"kubernetes.io/host-path/3e92427e-68a9-4496-9578-a0386bd5f5b3-dbus-socket\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.064622 master-1 kubenswrapper[4771]: I1011 10:48:57.064375 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5pt\" (UniqueName: \"kubernetes.io/projected/3e92427e-68a9-4496-9578-a0386bd5f5b3-kube-api-access-vl5pt\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.064622 master-1 kubenswrapper[4771]: I1011 10:48:57.064627 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovs-socket\" (UniqueName: \"kubernetes.io/host-path/3e92427e-68a9-4496-9578-a0386bd5f5b3-ovs-socket\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.064948 master-1 kubenswrapper[4771]: I1011 10:48:57.064683 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nmstate-lock\" (UniqueName: \"kubernetes.io/host-path/3e92427e-68a9-4496-9578-a0386bd5f5b3-nmstate-lock\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.065010 master-1 kubenswrapper[4771]: I1011 10:48:57.064961 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dbus-socket\" (UniqueName: \"kubernetes.io/host-path/3e92427e-68a9-4496-9578-a0386bd5f5b3-dbus-socket\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.079283 master-1 kubenswrapper[4771]: W1011 10:48:57.076812 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f01dc6_72cd_4eb0_8039_57150e0758bf.slice/crio-1d9eb93b1d5f75ed17266fb9b1595fe17751c138092994a9052144327e88854b WatchSource:0}: Error finding container 1d9eb93b1d5f75ed17266fb9b1595fe17751c138092994a9052144327e88854b: Status 404 returned error can't find the container with id 1d9eb93b1d5f75ed17266fb9b1595fe17751c138092994a9052144327e88854b Oct 11 10:48:57.103762 master-1 kubenswrapper[4771]: I1011 10:48:57.103674 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5pt\" (UniqueName: \"kubernetes.io/projected/3e92427e-68a9-4496-9578-a0386bd5f5b3-kube-api-access-vl5pt\") pod \"nmstate-handler-djsq6\" (UID: \"3e92427e-68a9-4496-9578-a0386bd5f5b3\") " pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.229158 master-1 kubenswrapper[4771]: I1011 10:48:57.229103 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:48:57.250218 master-1 kubenswrapper[4771]: W1011 10:48:57.249167 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e92427e_68a9_4496_9578_a0386bd5f5b3.slice/crio-d8122906cd222cfc295fa625a57fe54a4b6985be89018fdf7057dd4f32a54278 WatchSource:0}: Error finding container d8122906cd222cfc295fa625a57fe54a4b6985be89018fdf7057dd4f32a54278: Status 404 returned error can't find the container with id d8122906cd222cfc295fa625a57fe54a4b6985be89018fdf7057dd4f32a54278 Oct 11 10:48:57.301481 master-1 kubenswrapper[4771]: I1011 10:48:57.301423 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-console/console-69f8677c95-9ncnx"] Oct 11 10:48:57.302857 master-1 kubenswrapper[4771]: I1011 10:48:57.302817 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.306004 master-1 kubenswrapper[4771]: I1011 10:48:57.305585 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"console-oauth-config" Oct 11 10:48:57.306004 master-1 kubenswrapper[4771]: I1011 10:48:57.305618 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"console-dockercfg-8wmjp" Oct 11 10:48:57.306004 master-1 kubenswrapper[4771]: I1011 10:48:57.305589 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"console-config" Oct 11 10:48:57.306004 master-1 kubenswrapper[4771]: I1011 10:48:57.305779 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"oauth-serving-cert" Oct 11 10:48:57.306004 master-1 kubenswrapper[4771]: I1011 10:48:57.305995 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"service-ca" Oct 11 10:48:57.306912 master-1 kubenswrapper[4771]: I1011 10:48:57.306876 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-console"/"console-serving-cert" Oct 11 10:48:57.316546 master-1 kubenswrapper[4771]: I1011 10:48:57.316485 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f8677c95-9ncnx"] Oct 11 10:48:57.319456 master-1 kubenswrapper[4771]: I1011 10:48:57.319405 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-console"/"trusted-ca-bundle" Oct 11 10:48:57.369227 master-1 kubenswrapper[4771]: I1011 10:48:57.369151 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/733e6a5e-667b-4b9e-a359-577c976193f1-console-serving-cert\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.369227 master-1 kubenswrapper[4771]: I1011 10:48:57.369211 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-service-ca\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.369622 master-1 kubenswrapper[4771]: I1011 10:48:57.369263 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-oauth-serving-cert\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.369622 master-1 kubenswrapper[4771]: I1011 10:48:57.369291 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj645\" (UniqueName: \"kubernetes.io/projected/733e6a5e-667b-4b9e-a359-577c976193f1-kube-api-access-lj645\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.369622 master-1 kubenswrapper[4771]: I1011 10:48:57.369330 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-trusted-ca-bundle\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.369622 master-1 kubenswrapper[4771]: I1011 10:48:57.369381 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-console-config\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.369622 master-1 kubenswrapper[4771]: I1011 10:48:57.369414 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/733e6a5e-667b-4b9e-a359-577c976193f1-console-oauth-config\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.470210 master-1 kubenswrapper[4771]: I1011 10:48:57.470137 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-oauth-serving-cert\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.470210 master-1 kubenswrapper[4771]: I1011 10:48:57.470184 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-lj645\" (UniqueName: \"kubernetes.io/projected/733e6a5e-667b-4b9e-a359-577c976193f1-kube-api-access-lj645\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.470210 master-1 kubenswrapper[4771]: I1011 10:48:57.470227 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-trusted-ca-bundle\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.470658 master-1 kubenswrapper[4771]: I1011 10:48:57.470266 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-console-config\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.470658 master-1 kubenswrapper[4771]: I1011 10:48:57.470289 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/733e6a5e-667b-4b9e-a359-577c976193f1-console-oauth-config\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.470658 master-1 kubenswrapper[4771]: I1011 10:48:57.470318 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/733e6a5e-667b-4b9e-a359-577c976193f1-console-serving-cert\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.470658 master-1 kubenswrapper[4771]: I1011 10:48:57.470340 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-service-ca\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.471908 master-1 kubenswrapper[4771]: I1011 10:48:57.471750 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-oauth-serving-cert\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.472518 master-1 kubenswrapper[4771]: I1011 10:48:57.472454 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-console-config\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.472954 master-1 kubenswrapper[4771]: I1011 10:48:57.472918 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-trusted-ca-bundle\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.473125 master-1 kubenswrapper[4771]: I1011 10:48:57.473093 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/733e6a5e-667b-4b9e-a359-577c976193f1-service-ca\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.475478 master-1 kubenswrapper[4771]: I1011 10:48:57.475106 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/733e6a5e-667b-4b9e-a359-577c976193f1-console-serving-cert\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.476108 master-1 kubenswrapper[4771]: I1011 10:48:57.476012 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/733e6a5e-667b-4b9e-a359-577c976193f1-console-oauth-config\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.489862 master-1 kubenswrapper[4771]: I1011 10:48:57.489819 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj645\" (UniqueName: \"kubernetes.io/projected/733e6a5e-667b-4b9e-a359-577c976193f1-kube-api-access-lj645\") pod \"console-69f8677c95-9ncnx\" (UID: \"733e6a5e-667b-4b9e-a359-577c976193f1\") " pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.621959 master-1 kubenswrapper[4771]: I1011 10:48:57.621795 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:48:57.771091 master-1 kubenswrapper[4771]: I1011 10:48:57.771026 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-nmstate/nmstate-handler-djsq6" event={"ID":"3e92427e-68a9-4496-9578-a0386bd5f5b3","Type":"ContainerStarted","Data":"d8122906cd222cfc295fa625a57fe54a4b6985be89018fdf7057dd4f32a54278"} Oct 11 10:48:57.773314 master-1 kubenswrapper[4771]: I1011 10:48:57.773258 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/speaker-524kt" event={"ID":"72f01dc6-72cd-4eb0-8039-57150e0758bf","Type":"ContainerStarted","Data":"1d9eb93b1d5f75ed17266fb9b1595fe17751c138092994a9052144327e88854b"} Oct 11 10:48:58.117990 master-1 kubenswrapper[4771]: I1011 10:48:58.117937 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f8677c95-9ncnx"] Oct 11 10:48:58.237997 master-1 kubenswrapper[4771]: I1011 10:48:58.237943 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:48:58.238255 master-1 kubenswrapper[4771]: I1011 10:48:58.238011 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:48:59.788652 master-1 kubenswrapper[4771]: I1011 10:48:59.788610 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_42d61efaa0f96869cf2939026aad6022/kube-apiserver-cert-syncer/0.log" Oct 11 10:48:59.789265 master-1 kubenswrapper[4771]: I1011 10:48:59.789209 4771 generic.go:334] "Generic (PLEG): container finished" podID="42d61efaa0f96869cf2939026aad6022" containerID="d035b13d9431b1216e273c4ac7fb5eb87624d8740b70d29326082336302e3b46" exitCode=0 Oct 11 10:49:00.800220 master-1 kubenswrapper[4771]: I1011 10:49:00.800161 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f8677c95-9ncnx" event={"ID":"733e6a5e-667b-4b9e-a359-577c976193f1","Type":"ContainerStarted","Data":"aa3d533766c729d41f6c2e94ef952060ec8437e2048a8be7f1811e02f5315694"} Oct 11 10:49:01.446649 master-1 kubenswrapper[4771]: I1011 10:49:01.446605 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_42d61efaa0f96869cf2939026aad6022/kube-apiserver-cert-syncer/0.log" Oct 11 10:49:01.448694 master-1 kubenswrapper[4771]: I1011 10:49:01.448653 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:01.458194 master-1 kubenswrapper[4771]: I1011 10:49:01.458140 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="42d61efaa0f96869cf2939026aad6022" podUID="23141951a25391899fad7b9f2d5b6739" Oct 11 10:49:01.555050 master-1 kubenswrapper[4771]: I1011 10:49:01.552779 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-resource-dir\") pod \"42d61efaa0f96869cf2939026aad6022\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " Oct 11 10:49:01.555050 master-1 kubenswrapper[4771]: I1011 10:49:01.552899 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-cert-dir\") pod \"42d61efaa0f96869cf2939026aad6022\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " Oct 11 10:49:01.555050 master-1 kubenswrapper[4771]: I1011 10:49:01.552988 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-audit-dir\") pod \"42d61efaa0f96869cf2939026aad6022\" (UID: \"42d61efaa0f96869cf2939026aad6022\") " Oct 11 10:49:01.555050 master-1 kubenswrapper[4771]: I1011 10:49:01.553378 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-audit-dir" (OuterVolumeSpecName: "audit-dir") pod "42d61efaa0f96869cf2939026aad6022" (UID: "42d61efaa0f96869cf2939026aad6022"). InnerVolumeSpecName "audit-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:49:01.555050 master-1 kubenswrapper[4771]: I1011 10:49:01.553415 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-resource-dir" (OuterVolumeSpecName: "resource-dir") pod "42d61efaa0f96869cf2939026aad6022" (UID: "42d61efaa0f96869cf2939026aad6022"). InnerVolumeSpecName "resource-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:49:01.555050 master-1 kubenswrapper[4771]: I1011 10:49:01.553450 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-cert-dir" (OuterVolumeSpecName: "cert-dir") pod "42d61efaa0f96869cf2939026aad6022" (UID: "42d61efaa0f96869cf2939026aad6022"). InnerVolumeSpecName "cert-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:49:01.656545 master-1 kubenswrapper[4771]: I1011 10:49:01.655052 4771 reconciler_common.go:293] "Volume detached for volume \"audit-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-audit-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:49:01.656545 master-1 kubenswrapper[4771]: I1011 10:49:01.655093 4771 reconciler_common.go:293] "Volume detached for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-resource-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:49:01.656545 master-1 kubenswrapper[4771]: I1011 10:49:01.655102 4771 reconciler_common.go:293] "Volume detached for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/42d61efaa0f96869cf2939026aad6022-cert-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:49:01.814762 master-1 kubenswrapper[4771]: I1011 10:49:01.814690 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_42d61efaa0f96869cf2939026aad6022/kube-apiserver-cert-syncer/0.log" Oct 11 10:49:01.816568 master-1 kubenswrapper[4771]: I1011 10:49:01.815940 4771 scope.go:117] "RemoveContainer" containerID="a15e7539d2a0c42e8c6c8995bf98ff26ca0f322daf83394df48b4f13fc42d10b" Oct 11 10:49:01.816691 master-1 kubenswrapper[4771]: I1011 10:49:01.816653 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:01.822497 master-1 kubenswrapper[4771]: I1011 10:49:01.822458 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="42d61efaa0f96869cf2939026aad6022" podUID="23141951a25391899fad7b9f2d5b6739" Oct 11 10:49:01.839927 master-1 kubenswrapper[4771]: I1011 10:49:01.839209 4771 status_manager.go:861] "Pod was deleted and then recreated, skipping status update" pod="openshift-kube-apiserver/kube-apiserver-master-1" oldPodUID="42d61efaa0f96869cf2939026aad6022" podUID="23141951a25391899fad7b9f2d5b6739" Oct 11 10:49:02.448243 master-1 kubenswrapper[4771]: I1011 10:49:02.445903 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d61efaa0f96869cf2939026aad6022" path="/var/lib/kubelet/pods/42d61efaa0f96869cf2939026aad6022/volumes" Oct 11 10:49:02.449126 master-1 kubenswrapper[4771]: I1011 10:49:02.449043 4771 scope.go:117] "RemoveContainer" containerID="452189c1a156cff2357db3338f99f86d41c76ed0f97b4459672ad6a8fe0dc5c7" Oct 11 10:49:02.498612 master-1 kubenswrapper[4771]: I1011 10:49:02.498538 4771 scope.go:117] "RemoveContainer" containerID="55ecf6fefa862d92619ce534057ad20c836371d13f4c0d70468214b0bd6e3db4" Oct 11 10:49:02.535378 master-1 kubenswrapper[4771]: I1011 10:49:02.535322 4771 scope.go:117] "RemoveContainer" containerID="7e5a3711f36461fe4ced62a6738267cdf151c6f22d750936a4256bced2e89c2a" Oct 11 10:49:02.576406 master-1 kubenswrapper[4771]: I1011 10:49:02.575836 4771 scope.go:117] "RemoveContainer" containerID="d035b13d9431b1216e273c4ac7fb5eb87624d8740b70d29326082336302e3b46" Oct 11 10:49:02.641407 master-1 kubenswrapper[4771]: I1011 10:49:02.641375 4771 scope.go:117] "RemoveContainer" containerID="546001aeab4a76f01af18f5f0a0232cc48a20c2025802d7d9983eb8c840e0866" Oct 11 10:49:02.832588 master-1 kubenswrapper[4771]: I1011 10:49:02.832521 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-nmstate/nmstate-handler-djsq6" event={"ID":"3e92427e-68a9-4496-9578-a0386bd5f5b3","Type":"ContainerStarted","Data":"82b3117028c1393abaa55457d166e03f4b9b033326efa34c5fda13a687ec71ed"} Oct 11 10:49:02.833289 master-1 kubenswrapper[4771]: I1011 10:49:02.832616 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:49:02.836239 master-1 kubenswrapper[4771]: I1011 10:49:02.836185 4771 generic.go:334] "Generic (PLEG): container finished" podID="f1e63653-b356-4bf6-b91a-6d386b1f3c33" containerID="061c58050d82ba1bcc1df769608d0f7dd7a5b57018512b29a7cd4a8db046629f" exitCode=0 Oct 11 10:49:02.836389 master-1 kubenswrapper[4771]: I1011 10:49:02.836263 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerDied","Data":"061c58050d82ba1bcc1df769608d0f7dd7a5b57018512b29a7cd4a8db046629f"} Oct 11 10:49:02.838730 master-1 kubenswrapper[4771]: I1011 10:49:02.838688 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/speaker-524kt" event={"ID":"72f01dc6-72cd-4eb0-8039-57150e0758bf","Type":"ContainerStarted","Data":"13f878e8ffe2bb81a2ff9441a77dd5dc7f6643745de0f5325acf5d0ca8d6ca26"} Oct 11 10:49:02.841571 master-1 kubenswrapper[4771]: I1011 10:49:02.841540 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f8677c95-9ncnx" event={"ID":"733e6a5e-667b-4b9e-a359-577c976193f1","Type":"ContainerStarted","Data":"d0a0690f05bf6419109c3dbe54649f8b09999cb9e8c0e0810ef25797a35afea4"} Oct 11 10:49:02.861543 master-1 kubenswrapper[4771]: I1011 10:49:02.861440 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-nmstate/nmstate-handler-djsq6" podStartSLOduration=1.59400261 podStartE2EDuration="6.86142442s" podCreationTimestamp="2025-10-11 10:48:56 +0000 UTC" firstStartedPulling="2025-10-11 10:48:57.252868565 +0000 UTC m=+1369.227095016" lastFinishedPulling="2025-10-11 10:49:02.520290385 +0000 UTC m=+1374.494516826" observedRunningTime="2025-10-11 10:49:02.859536395 +0000 UTC m=+1374.833762836" watchObservedRunningTime="2025-10-11 10:49:02.86142442 +0000 UTC m=+1374.835650871" Oct 11 10:49:02.918929 master-1 kubenswrapper[4771]: I1011 10:49:02.918840 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-69f8677c95-9ncnx" podStartSLOduration=5.918818183 podStartE2EDuration="5.918818183s" podCreationTimestamp="2025-10-11 10:48:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:49:02.916002631 +0000 UTC m=+1374.890229072" watchObservedRunningTime="2025-10-11 10:49:02.918818183 +0000 UTC m=+1374.893044624" Oct 11 10:49:03.238194 master-1 kubenswrapper[4771]: I1011 10:49:03.237556 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:49:03.238194 master-1 kubenswrapper[4771]: I1011 10:49:03.237637 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:49:03.853458 master-1 kubenswrapper[4771]: I1011 10:49:03.853267 4771 generic.go:334] "Generic (PLEG): container finished" podID="f1e63653-b356-4bf6-b91a-6d386b1f3c33" containerID="0334ffbda62413275839f28919de6c11cdbed4cb055f314d0f4b773c855eb34c" exitCode=0 Oct 11 10:49:03.853458 master-1 kubenswrapper[4771]: I1011 10:49:03.853410 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerDied","Data":"0334ffbda62413275839f28919de6c11cdbed4cb055f314d0f4b773c855eb34c"} Oct 11 10:49:04.864583 master-1 kubenswrapper[4771]: I1011 10:49:04.864509 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerDied","Data":"d934544b69458e648d28108ff72e6e855ddc5b36170c4331901f85f9b2013df3"} Oct 11 10:49:04.865202 master-1 kubenswrapper[4771]: I1011 10:49:04.864401 4771 generic.go:334] "Generic (PLEG): container finished" podID="f1e63653-b356-4bf6-b91a-6d386b1f3c33" containerID="d934544b69458e648d28108ff72e6e855ddc5b36170c4331901f85f9b2013df3" exitCode=0 Oct 11 10:49:04.884126 master-1 kubenswrapper[4771]: I1011 10:49:04.877502 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/speaker-524kt" event={"ID":"72f01dc6-72cd-4eb0-8039-57150e0758bf","Type":"ContainerStarted","Data":"d37b1e37373baf43f3f67ce871913965475fa1ee724fe4135d14b88b1c10ebba"} Oct 11 10:49:04.884126 master-1 kubenswrapper[4771]: I1011 10:49:04.879006 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="metallb-system/speaker-524kt" Oct 11 10:49:04.932129 master-1 kubenswrapper[4771]: I1011 10:49:04.930900 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="metallb-system/speaker-524kt" podStartSLOduration=4.412853247 podStartE2EDuration="10.930883435s" podCreationTimestamp="2025-10-11 10:48:54 +0000 UTC" firstStartedPulling="2025-10-11 10:48:57.083822787 +0000 UTC m=+1369.058049258" lastFinishedPulling="2025-10-11 10:49:03.601853005 +0000 UTC m=+1375.576079446" observedRunningTime="2025-10-11 10:49:04.929979558 +0000 UTC m=+1376.904205999" watchObservedRunningTime="2025-10-11 10:49:04.930883435 +0000 UTC m=+1376.905109876" Oct 11 10:49:05.896200 master-1 kubenswrapper[4771]: I1011 10:49:05.896120 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerStarted","Data":"06797530c40fd5883bac35a7432b5b6b13c75d2f48ebb82338c56a27a05cc991"} Oct 11 10:49:05.896200 master-1 kubenswrapper[4771]: I1011 10:49:05.896199 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerStarted","Data":"6656112c9be2dff2eeb0f857ebb0a4ef3d7f1d0a1e0b9003159fc88ccbdf3b65"} Oct 11 10:49:05.897281 master-1 kubenswrapper[4771]: I1011 10:49:05.896224 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerStarted","Data":"d4906bccb5f00bc5ad9a090e2b636be0c07ebd66559498406de5aa2630b2b54d"} Oct 11 10:49:06.914549 master-1 kubenswrapper[4771]: I1011 10:49:06.914451 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerStarted","Data":"8a3c45be82c19ad78af58745d6ecb1937c5ad168c9d253338098a26c219e7bd0"} Oct 11 10:49:06.915681 master-1 kubenswrapper[4771]: I1011 10:49:06.915610 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:49:06.915992 master-1 kubenswrapper[4771]: I1011 10:49:06.915913 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerStarted","Data":"f6bc0bac91103a3736bac8cdadc3f57883d3d35f6390a75d1223cb724e09ca3d"} Oct 11 10:49:06.916295 master-1 kubenswrapper[4771]: I1011 10:49:06.916220 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="metallb-system/frr-k8s-lvzhx" event={"ID":"f1e63653-b356-4bf6-b91a-6d386b1f3c33","Type":"ContainerStarted","Data":"ce536cb60007c1de48f428cc06d44c25cbb4e9e6d2b2e852c371a1ae4856ad23"} Oct 11 10:49:06.955752 master-1 kubenswrapper[4771]: I1011 10:49:06.952259 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="metallb-system/frr-k8s-lvzhx" podStartSLOduration=6.434047858 podStartE2EDuration="13.952229776s" podCreationTimestamp="2025-10-11 10:48:53 +0000 UTC" firstStartedPulling="2025-10-11 10:48:54.930977546 +0000 UTC m=+1366.905203997" lastFinishedPulling="2025-10-11 10:49:02.449159474 +0000 UTC m=+1374.423385915" observedRunningTime="2025-10-11 10:49:06.950032642 +0000 UTC m=+1378.924259163" watchObservedRunningTime="2025-10-11 10:49:06.952229776 +0000 UTC m=+1378.926456257" Oct 11 10:49:07.259047 master-1 kubenswrapper[4771]: I1011 10:49:07.258898 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-nmstate/nmstate-handler-djsq6" Oct 11 10:49:07.622402 master-1 kubenswrapper[4771]: I1011 10:49:07.622283 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:49:07.622808 master-1 kubenswrapper[4771]: I1011 10:49:07.622437 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:49:07.631395 master-1 kubenswrapper[4771]: I1011 10:49:07.631308 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:49:07.946542 master-1 kubenswrapper[4771]: I1011 10:49:07.945770 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69f8677c95-9ncnx" Oct 11 10:49:08.238518 master-1 kubenswrapper[4771]: I1011 10:49:08.238394 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:49:08.238518 master-1 kubenswrapper[4771]: I1011 10:49:08.238476 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:49:09.503572 master-1 kubenswrapper[4771]: I1011 10:49:09.503218 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-etcd/installer-5-master-1"] Oct 11 10:49:09.512798 master-1 kubenswrapper[4771]: I1011 10:49:09.512722 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-etcd/installer-5-master-1"] Oct 11 10:49:09.758001 master-1 kubenswrapper[4771]: I1011 10:49:09.757838 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:49:09.795772 master-1 kubenswrapper[4771]: I1011 10:49:09.795676 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:49:10.449474 master-1 kubenswrapper[4771]: I1011 10:49:10.449386 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0f830cc-d36c-4ccd-97cb-2d4a99726684" path="/var/lib/kubelet/pods/f0f830cc-d36c-4ccd-97cb-2d4a99726684/volumes" Oct 11 10:49:12.436693 master-1 kubenswrapper[4771]: I1011 10:49:12.436604 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:12.464963 master-1 kubenswrapper[4771]: I1011 10:49:12.464896 4771 kubelet.go:1909] "Trying to delete pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="dd9c0979-c66d-405d-b41b-02fec7c5a5da" Oct 11 10:49:12.464963 master-1 kubenswrapper[4771]: I1011 10:49:12.464954 4771 mirror_client.go:130] "Deleting a mirror pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" podUID="dd9c0979-c66d-405d-b41b-02fec7c5a5da" Oct 11 10:49:12.492901 master-1 kubenswrapper[4771]: I1011 10:49:12.492781 4771 kubelet.go:1914] "Deleted mirror pod because it is outdated" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:12.493793 master-1 kubenswrapper[4771]: I1011 10:49:12.493712 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:49:12.508135 master-1 kubenswrapper[4771]: I1011 10:49:12.508064 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:49:12.519099 master-1 kubenswrapper[4771]: I1011 10:49:12.519042 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:12.524428 master-1 kubenswrapper[4771]: I1011 10:49:12.524349 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/kube-apiserver-master-1"] Oct 11 10:49:12.549997 master-1 kubenswrapper[4771]: W1011 10:49:12.549897 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23141951a25391899fad7b9f2d5b6739.slice/crio-96af06f738dd615b275d7cdbc3ed2324efae9eec171f4d91d3b8d40221367da9 WatchSource:0}: Error finding container 96af06f738dd615b275d7cdbc3ed2324efae9eec171f4d91d3b8d40221367da9: Status 404 returned error can't find the container with id 96af06f738dd615b275d7cdbc3ed2324efae9eec171f4d91d3b8d40221367da9 Oct 11 10:49:12.969469 master-1 kubenswrapper[4771]: I1011 10:49:12.969403 4771 generic.go:334] "Generic (PLEG): container finished" podID="23141951a25391899fad7b9f2d5b6739" containerID="5e79b10186bcde1683e0ff15f9bb04b0604789f489cf69d5e42d464f0d1d0aed" exitCode=0 Oct 11 10:49:12.969469 master-1 kubenswrapper[4771]: I1011 10:49:12.969468 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"23141951a25391899fad7b9f2d5b6739","Type":"ContainerDied","Data":"5e79b10186bcde1683e0ff15f9bb04b0604789f489cf69d5e42d464f0d1d0aed"} Oct 11 10:49:12.969901 master-1 kubenswrapper[4771]: I1011 10:49:12.969507 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"23141951a25391899fad7b9f2d5b6739","Type":"ContainerStarted","Data":"96af06f738dd615b275d7cdbc3ed2324efae9eec171f4d91d3b8d40221367da9"} Oct 11 10:49:13.238499 master-1 kubenswrapper[4771]: I1011 10:49:13.238408 4771 patch_prober.go:28] interesting pod/kube-apiserver-guard-master-1 container/guard namespace/openshift-kube-apiserver: Readiness probe status=failure output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" start-of-body= Oct 11 10:49:13.238786 master-1 kubenswrapper[4771]: I1011 10:49:13.238496 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" podUID="86b914fa-4ccd-42fb-965a-a1bc19442489" containerName="guard" probeResult="failure" output="Get \"https://192.168.34.11:6443/readyz\": dial tcp 192.168.34.11:6443: connect: connection refused" Oct 11 10:49:13.722310 master-1 kubenswrapper[4771]: I1011 10:49:13.722238 4771 scope.go:117] "RemoveContainer" containerID="2b7fb64c483453dbfbd93869288690ed38d6d29cb105ac6ec22c06d0d9551aa1" Oct 11 10:49:13.994465 master-1 kubenswrapper[4771]: I1011 10:49:13.994347 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"23141951a25391899fad7b9f2d5b6739","Type":"ContainerStarted","Data":"6981b4508ac74e45d282fe5a59f106dabfea7c7c3293de729e9d6f0db68c7510"} Oct 11 10:49:13.994465 master-1 kubenswrapper[4771]: I1011 10:49:13.994465 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"23141951a25391899fad7b9f2d5b6739","Type":"ContainerStarted","Data":"07c0fc0bccb7b7da4962275bebaa5f5ab1861795bc9d86159b3e163ce05cddac"} Oct 11 10:49:13.994832 master-1 kubenswrapper[4771]: I1011 10:49:13.994490 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"23141951a25391899fad7b9f2d5b6739","Type":"ContainerStarted","Data":"78130b5301fab8a631d47c753616bc63c2d7d8f2931253367a84eb0adf8538fb"} Oct 11 10:49:14.764532 master-1 kubenswrapper[4771]: I1011 10:49:14.760986 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="metallb-system/frr-k8s-lvzhx" Oct 11 10:49:15.004995 master-1 kubenswrapper[4771]: I1011 10:49:15.004941 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"23141951a25391899fad7b9f2d5b6739","Type":"ContainerStarted","Data":"761b6c7301a42f97ce8830816995738e797ac6ef0b6d2847f3ce0b1a956aa630"} Oct 11 10:49:15.004995 master-1 kubenswrapper[4771]: I1011 10:49:15.004998 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/kube-apiserver-master-1" event={"ID":"23141951a25391899fad7b9f2d5b6739","Type":"ContainerStarted","Data":"6b790055ab747dc39b0de5102915f7f68179102f3a383ae626b746d0c22b0f1f"} Oct 11 10:49:15.005256 master-1 kubenswrapper[4771]: I1011 10:49:15.005177 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:17.053735 master-1 kubenswrapper[4771]: I1011 10:49:17.053636 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="metallb-system/speaker-524kt" Oct 11 10:49:17.079663 master-1 kubenswrapper[4771]: I1011 10:49:17.079551 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/kube-apiserver-master-1" podStartSLOduration=5.079524857 podStartE2EDuration="5.079524857s" podCreationTimestamp="2025-10-11 10:49:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:49:15.30612152 +0000 UTC m=+1387.280347971" watchObservedRunningTime="2025-10-11 10:49:17.079524857 +0000 UTC m=+1389.053751338" Oct 11 10:49:17.520434 master-1 kubenswrapper[4771]: I1011 10:49:17.520224 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:17.520434 master-1 kubenswrapper[4771]: I1011 10:49:17.520289 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:17.533787 master-1 kubenswrapper[4771]: I1011 10:49:17.533707 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:18.036692 master-1 kubenswrapper[4771]: I1011 10:49:18.036611 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:18.245177 master-1 kubenswrapper[4771]: I1011 10:49:18.245102 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-guard-master-1" Oct 11 10:49:26.301470 master-1 kubenswrapper[4771]: I1011 10:49:26.301330 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-storage/vg-manager-l9x5s"] Oct 11 10:49:26.303118 master-1 kubenswrapper[4771]: I1011 10:49:26.303084 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.307544 master-1 kubenswrapper[4771]: I1011 10:49:26.307471 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-storage"/"openshift-service-ca.crt" Oct 11 10:49:26.308474 master-1 kubenswrapper[4771]: I1011 10:49:26.307607 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-storage"/"kube-root-ca.crt" Oct 11 10:49:26.308474 master-1 kubenswrapper[4771]: I1011 10:49:26.307503 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-storage"/"vg-manager-metrics-cert" Oct 11 10:49:26.331664 master-1 kubenswrapper[4771]: I1011 10:49:26.329717 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-storage/vg-manager-l9x5s"] Oct 11 10:49:26.409568 master-1 kubenswrapper[4771]: I1011 10:49:26.409460 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pod-volumes-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-pod-volumes-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.409938 master-1 kubenswrapper[4771]: I1011 10:49:26.409630 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-metrics-cert\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.409938 master-1 kubenswrapper[4771]: I1011 10:49:26.409686 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-device-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.409938 master-1 kubenswrapper[4771]: I1011 10:49:26.409735 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-udev\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-run-udev\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.409938 master-1 kubenswrapper[4771]: I1011 10:49:26.409832 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file-lock-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-file-lock-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.409938 master-1 kubenswrapper[4771]: I1011 10:49:26.409884 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-registration-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.409938 master-1 kubenswrapper[4771]: I1011 10:49:26.409926 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-plugin-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-node-plugin-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.410427 master-1 kubenswrapper[4771]: I1011 10:49:26.409980 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lvmd-config\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-lvmd-config\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.410427 master-1 kubenswrapper[4771]: I1011 10:49:26.410040 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-sys\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.410427 master-1 kubenswrapper[4771]: I1011 10:49:26.410126 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"csi-plugin-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-csi-plugin-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.410427 master-1 kubenswrapper[4771]: I1011 10:49:26.410187 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6w2m\" (UniqueName: \"kubernetes.io/projected/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-kube-api-access-v6w2m\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.511804 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pod-volumes-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-pod-volumes-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512008 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-metrics-cert\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512076 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-device-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512005 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pod-volumes-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-pod-volumes-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512160 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-udev\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-run-udev\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512217 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"file-lock-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-file-lock-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512268 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-registration-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512279 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-udev\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-run-udev\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512315 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"node-plugin-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-node-plugin-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512334 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-device-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512410 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"lvmd-config\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-lvmd-config\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512503 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-sys\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512580 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"csi-plugin-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-csi-plugin-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512656 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-v6w2m\" (UniqueName: \"kubernetes.io/projected/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-kube-api-access-v6w2m\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512683 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"file-lock-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-file-lock-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512716 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-registration-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512925 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-sys\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.514941 master-1 kubenswrapper[4771]: I1011 10:49:26.512946 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"lvmd-config\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-lvmd-config\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.516522 master-1 kubenswrapper[4771]: I1011 10:49:26.512969 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"node-plugin-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-node-plugin-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.516522 master-1 kubenswrapper[4771]: I1011 10:49:26.513093 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"csi-plugin-dir\" (UniqueName: \"kubernetes.io/host-path/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-csi-plugin-dir\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.517698 master-1 kubenswrapper[4771]: I1011 10:49:26.517642 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-metrics-cert\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.535149 master-1 kubenswrapper[4771]: I1011 10:49:26.535074 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6w2m\" (UniqueName: \"kubernetes.io/projected/5f47a805-6ec5-4d90-a1be-dfaec7c5c818-kube-api-access-v6w2m\") pod \"vg-manager-l9x5s\" (UID: \"5f47a805-6ec5-4d90-a1be-dfaec7c5c818\") " pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:26.638161 master-1 kubenswrapper[4771]: I1011 10:49:26.637966 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:27.113753 master-1 kubenswrapper[4771]: I1011 10:49:27.113671 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-storage/vg-manager-l9x5s"] Oct 11 10:49:27.121383 master-1 kubenswrapper[4771]: W1011 10:49:27.121283 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f47a805_6ec5_4d90_a1be_dfaec7c5c818.slice/crio-d9862099bb13f3420abdd504122ac2ec077e7f887c15a2f7115b60bd375a4c7b WatchSource:0}: Error finding container d9862099bb13f3420abdd504122ac2ec077e7f887c15a2f7115b60bd375a4c7b: Status 404 returned error can't find the container with id d9862099bb13f3420abdd504122ac2ec077e7f887c15a2f7115b60bd375a4c7b Oct 11 10:49:28.112013 master-1 kubenswrapper[4771]: I1011 10:49:28.111650 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-storage/vg-manager-l9x5s" event={"ID":"5f47a805-6ec5-4d90-a1be-dfaec7c5c818","Type":"ContainerStarted","Data":"d9862099bb13f3420abdd504122ac2ec077e7f887c15a2f7115b60bd375a4c7b"} Oct 11 10:49:32.138936 master-1 kubenswrapper[4771]: I1011 10:49:32.138750 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-storage/vg-manager-l9x5s" event={"ID":"5f47a805-6ec5-4d90-a1be-dfaec7c5c818","Type":"ContainerStarted","Data":"d8938647ff6d38271de4e8bdb38b410218398c6b3c96d71c30f2ceb2a91f05cf"} Oct 11 10:49:32.352565 master-1 kubenswrapper[4771]: I1011 10:49:32.352474 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-storage/vg-manager-l9x5s" podStartSLOduration=1.844631008 podStartE2EDuration="6.352453918s" podCreationTimestamp="2025-10-11 10:49:26 +0000 UTC" firstStartedPulling="2025-10-11 10:49:27.124309256 +0000 UTC m=+1399.098535727" lastFinishedPulling="2025-10-11 10:49:31.632132186 +0000 UTC m=+1403.606358637" observedRunningTime="2025-10-11 10:49:32.350842842 +0000 UTC m=+1404.325069303" watchObservedRunningTime="2025-10-11 10:49:32.352453918 +0000 UTC m=+1404.326680379" Oct 11 10:49:32.526778 master-1 kubenswrapper[4771]: I1011 10:49:32.526617 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-apiserver/kube-apiserver-master-1" Oct 11 10:49:35.163757 master-1 kubenswrapper[4771]: I1011 10:49:35.163660 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-storage_vg-manager-l9x5s_5f47a805-6ec5-4d90-a1be-dfaec7c5c818/vg-manager/0.log" Oct 11 10:49:35.164807 master-1 kubenswrapper[4771]: I1011 10:49:35.163758 4771 generic.go:334] "Generic (PLEG): container finished" podID="5f47a805-6ec5-4d90-a1be-dfaec7c5c818" containerID="d8938647ff6d38271de4e8bdb38b410218398c6b3c96d71c30f2ceb2a91f05cf" exitCode=1 Oct 11 10:49:35.164807 master-1 kubenswrapper[4771]: I1011 10:49:35.163814 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-storage/vg-manager-l9x5s" event={"ID":"5f47a805-6ec5-4d90-a1be-dfaec7c5c818","Type":"ContainerDied","Data":"d8938647ff6d38271de4e8bdb38b410218398c6b3c96d71c30f2ceb2a91f05cf"} Oct 11 10:49:35.164807 master-1 kubenswrapper[4771]: I1011 10:49:35.164527 4771 scope.go:117] "RemoveContainer" containerID="d8938647ff6d38271de4e8bdb38b410218398c6b3c96d71c30f2ceb2a91f05cf" Oct 11 10:49:35.577242 master-1 kubenswrapper[4771]: I1011 10:49:35.577153 4771 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/topolvm.io-reg.sock" Oct 11 10:49:36.175766 master-1 kubenswrapper[4771]: I1011 10:49:36.175557 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-storage_vg-manager-l9x5s_5f47a805-6ec5-4d90-a1be-dfaec7c5c818/vg-manager/0.log" Oct 11 10:49:36.175766 master-1 kubenswrapper[4771]: I1011 10:49:36.175638 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-storage/vg-manager-l9x5s" event={"ID":"5f47a805-6ec5-4d90-a1be-dfaec7c5c818","Type":"ContainerStarted","Data":"5affce9a11708ad0d32c9ea6232aa6aff06c771b15c6fdae9dd419b025450ff7"} Oct 11 10:49:36.392646 master-1 kubenswrapper[4771]: I1011 10:49:36.392314 4771 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/topolvm.io-reg.sock","Timestamp":"2025-10-11T10:49:35.577190949Z","Handler":null,"Name":""} Oct 11 10:49:36.396210 master-1 kubenswrapper[4771]: I1011 10:49:36.396139 4771 csi_plugin.go:100] kubernetes.io/csi: Trying to validate a new CSI Driver with name: topolvm.io endpoint: /var/lib/kubelet/plugins/topolvm.io/node/csi-topolvm.sock versions: 1.0.0 Oct 11 10:49:36.396408 master-1 kubenswrapper[4771]: I1011 10:49:36.396254 4771 csi_plugin.go:113] kubernetes.io/csi: Register new plugin with name: topolvm.io at endpoint: /var/lib/kubelet/plugins/topolvm.io/node/csi-topolvm.sock Oct 11 10:49:36.638910 master-1 kubenswrapper[4771]: I1011 10:49:36.638799 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:46.642662 master-1 kubenswrapper[4771]: I1011 10:49:46.642579 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:46.643609 master-1 kubenswrapper[4771]: I1011 10:49:46.643071 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:49:46.644108 master-1 kubenswrapper[4771]: I1011 10:49:46.644061 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-storage/vg-manager-l9x5s" Oct 11 10:50:05.806787 master-1 kubenswrapper[4771]: I1011 10:50:05.806636 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-kube-apiserver/revision-pruner-6-master-1"] Oct 11 10:50:05.809592 master-1 kubenswrapper[4771]: I1011 10:50:05.809186 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:05.817849 master-1 kubenswrapper[4771]: I1011 10:50:05.817771 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-kube-apiserver"/"installer-sa-dockercfg-t28rg" Oct 11 10:50:05.828030 master-1 kubenswrapper[4771]: I1011 10:50:05.827968 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/revision-pruner-6-master-1"] Oct 11 10:50:05.862847 master-1 kubenswrapper[4771]: I1011 10:50:05.862732 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c72a1cdf-d776-4854-b379-bde17097bab0-kube-api-access\") pod \"revision-pruner-6-master-1\" (UID: \"c72a1cdf-d776-4854-b379-bde17097bab0\") " pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:05.863124 master-1 kubenswrapper[4771]: I1011 10:50:05.862884 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c72a1cdf-d776-4854-b379-bde17097bab0-kubelet-dir\") pod \"revision-pruner-6-master-1\" (UID: \"c72a1cdf-d776-4854-b379-bde17097bab0\") " pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:05.965304 master-1 kubenswrapper[4771]: I1011 10:50:05.964721 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c72a1cdf-d776-4854-b379-bde17097bab0-kube-api-access\") pod \"revision-pruner-6-master-1\" (UID: \"c72a1cdf-d776-4854-b379-bde17097bab0\") " pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:05.965713 master-1 kubenswrapper[4771]: I1011 10:50:05.965415 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c72a1cdf-d776-4854-b379-bde17097bab0-kubelet-dir\") pod \"revision-pruner-6-master-1\" (UID: \"c72a1cdf-d776-4854-b379-bde17097bab0\") " pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:05.965713 master-1 kubenswrapper[4771]: I1011 10:50:05.965563 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c72a1cdf-d776-4854-b379-bde17097bab0-kubelet-dir\") pod \"revision-pruner-6-master-1\" (UID: \"c72a1cdf-d776-4854-b379-bde17097bab0\") " pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:05.985838 master-1 kubenswrapper[4771]: I1011 10:50:05.985707 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c72a1cdf-d776-4854-b379-bde17097bab0-kube-api-access\") pod \"revision-pruner-6-master-1\" (UID: \"c72a1cdf-d776-4854-b379-bde17097bab0\") " pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:06.133910 master-1 kubenswrapper[4771]: I1011 10:50:06.133759 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:06.595219 master-1 kubenswrapper[4771]: I1011 10:50:06.595018 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-kube-apiserver/revision-pruner-6-master-1"] Oct 11 10:50:07.435231 master-1 kubenswrapper[4771]: I1011 10:50:07.435120 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/revision-pruner-6-master-1" event={"ID":"c72a1cdf-d776-4854-b379-bde17097bab0","Type":"ContainerStarted","Data":"f5834e38fc462bced57a8335af6c00a64ee6e2c61e99884fc618f9c7f277f266"} Oct 11 10:50:07.435231 master-1 kubenswrapper[4771]: I1011 10:50:07.435209 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/revision-pruner-6-master-1" event={"ID":"c72a1cdf-d776-4854-b379-bde17097bab0","Type":"ContainerStarted","Data":"bc4363b6b2838fc8c75a54bdd92f428b47f1f3dc2a5fc935fbfd0bcc098edabf"} Oct 11 10:50:07.460166 master-1 kubenswrapper[4771]: I1011 10:50:07.460041 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-apiserver/revision-pruner-6-master-1" podStartSLOduration=2.4600203130000002 podStartE2EDuration="2.460020313s" podCreationTimestamp="2025-10-11 10:50:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:50:07.456948714 +0000 UTC m=+1439.431175165" watchObservedRunningTime="2025-10-11 10:50:07.460020313 +0000 UTC m=+1439.434246764" Oct 11 10:50:08.444127 master-1 kubenswrapper[4771]: I1011 10:50:08.444023 4771 generic.go:334] "Generic (PLEG): container finished" podID="c72a1cdf-d776-4854-b379-bde17097bab0" containerID="f5834e38fc462bced57a8335af6c00a64ee6e2c61e99884fc618f9c7f277f266" exitCode=0 Oct 11 10:50:08.451703 master-1 kubenswrapper[4771]: I1011 10:50:08.451597 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/revision-pruner-6-master-1" event={"ID":"c72a1cdf-d776-4854-b379-bde17097bab0","Type":"ContainerDied","Data":"f5834e38fc462bced57a8335af6c00a64ee6e2c61e99884fc618f9c7f277f266"} Oct 11 10:50:09.919092 master-1 kubenswrapper[4771]: I1011 10:50:09.918978 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:10.031269 master-1 kubenswrapper[4771]: I1011 10:50:10.031136 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c72a1cdf-d776-4854-b379-bde17097bab0-kubelet-dir\") pod \"c72a1cdf-d776-4854-b379-bde17097bab0\" (UID: \"c72a1cdf-d776-4854-b379-bde17097bab0\") " Oct 11 10:50:10.031789 master-1 kubenswrapper[4771]: I1011 10:50:10.031333 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c72a1cdf-d776-4854-b379-bde17097bab0-kube-api-access\") pod \"c72a1cdf-d776-4854-b379-bde17097bab0\" (UID: \"c72a1cdf-d776-4854-b379-bde17097bab0\") " Oct 11 10:50:10.031789 master-1 kubenswrapper[4771]: I1011 10:50:10.031378 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/c72a1cdf-d776-4854-b379-bde17097bab0-kubelet-dir" (OuterVolumeSpecName: "kubelet-dir") pod "c72a1cdf-d776-4854-b379-bde17097bab0" (UID: "c72a1cdf-d776-4854-b379-bde17097bab0"). InnerVolumeSpecName "kubelet-dir". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:50:10.032074 master-1 kubenswrapper[4771]: I1011 10:50:10.031949 4771 reconciler_common.go:293] "Volume detached for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c72a1cdf-d776-4854-b379-bde17097bab0-kubelet-dir\") on node \"master-1\" DevicePath \"\"" Oct 11 10:50:10.043721 master-1 kubenswrapper[4771]: I1011 10:50:10.043649 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72a1cdf-d776-4854-b379-bde17097bab0-kube-api-access" (OuterVolumeSpecName: "kube-api-access") pod "c72a1cdf-d776-4854-b379-bde17097bab0" (UID: "c72a1cdf-d776-4854-b379-bde17097bab0"). InnerVolumeSpecName "kube-api-access". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:50:10.133947 master-1 kubenswrapper[4771]: I1011 10:50:10.133747 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access\" (UniqueName: \"kubernetes.io/projected/c72a1cdf-d776-4854-b379-bde17097bab0-kube-api-access\") on node \"master-1\" DevicePath \"\"" Oct 11 10:50:10.240701 master-1 kubenswrapper[4771]: I1011 10:50:10.240615 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-kube-apiserver/installer-1-master-1"] Oct 11 10:50:10.268656 master-1 kubenswrapper[4771]: I1011 10:50:10.268591 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-kube-apiserver/installer-1-master-1"] Oct 11 10:50:10.454050 master-1 kubenswrapper[4771]: I1011 10:50:10.453842 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e7e1ec-47a8-4283-9119-0d9d1343963e" path="/var/lib/kubelet/pods/f5e7e1ec-47a8-4283-9119-0d9d1343963e/volumes" Oct 11 10:50:10.466931 master-1 kubenswrapper[4771]: I1011 10:50:10.463488 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-kube-apiserver/revision-pruner-6-master-1" event={"ID":"c72a1cdf-d776-4854-b379-bde17097bab0","Type":"ContainerDied","Data":"bc4363b6b2838fc8c75a54bdd92f428b47f1f3dc2a5fc935fbfd0bcc098edabf"} Oct 11 10:50:10.466931 master-1 kubenswrapper[4771]: I1011 10:50:10.463561 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc4363b6b2838fc8c75a54bdd92f428b47f1f3dc2a5fc935fbfd0bcc098edabf" Oct 11 10:50:10.466931 master-1 kubenswrapper[4771]: I1011 10:50:10.463650 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-apiserver/revision-pruner-6-master-1" Oct 11 10:50:13.828630 master-1 kubenswrapper[4771]: I1011 10:50:13.828503 4771 scope.go:117] "RemoveContainer" containerID="d38cc7e81ae0071969a185999498646cddc10ee8b65bed60da29b4c1f46a55dc" Oct 11 10:51:56.872827 master-1 kubenswrapper[4771]: I1011 10:51:56.872730 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76"] Oct 11 10:51:56.877269 master-1 kubenswrapper[4771]: E1011 10:51:56.873083 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="c72a1cdf-d776-4854-b379-bde17097bab0" containerName="pruner" Oct 11 10:51:56.877269 master-1 kubenswrapper[4771]: I1011 10:51:56.873100 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72a1cdf-d776-4854-b379-bde17097bab0" containerName="pruner" Oct 11 10:51:56.877269 master-1 kubenswrapper[4771]: I1011 10:51:56.873209 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="c72a1cdf-d776-4854-b379-bde17097bab0" containerName="pruner" Oct 11 10:51:56.877269 master-1 kubenswrapper[4771]: I1011 10:51:56.874081 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" Oct 11 10:51:56.881710 master-1 kubenswrapper[4771]: I1011 10:51:56.881633 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack-operators"/"openshift-service-ca.crt" Oct 11 10:51:56.882816 master-1 kubenswrapper[4771]: I1011 10:51:56.881694 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack-operators"/"kube-root-ca.crt" Oct 11 10:51:56.908424 master-1 kubenswrapper[4771]: I1011 10:51:56.908341 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76"] Oct 11 10:51:56.918914 master-1 kubenswrapper[4771]: I1011 10:51:56.918854 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnmnt\" (UniqueName: \"kubernetes.io/projected/608a645c-104f-4fab-b1c6-cbcae70ca0f4-kube-api-access-pnmnt\") pod \"heat-operator-controller-manager-68fc865f87-dfx76\" (UID: \"608a645c-104f-4fab-b1c6-cbcae70ca0f4\") " pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" Oct 11 10:51:56.989184 master-1 kubenswrapper[4771]: I1011 10:51:56.989102 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p"] Oct 11 10:51:56.992375 master-1 kubenswrapper[4771]: I1011 10:51:56.992322 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" Oct 11 10:51:57.008717 master-1 kubenswrapper[4771]: I1011 10:51:57.008631 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p"] Oct 11 10:51:57.020234 master-1 kubenswrapper[4771]: I1011 10:51:57.020143 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqpww\" (UniqueName: \"kubernetes.io/projected/9f700fad-87cd-467c-9b8a-99a82dd72d9b-kube-api-access-mqpww\") pod \"ironic-operator-controller-manager-6b498574d4-brh6p\" (UID: \"9f700fad-87cd-467c-9b8a-99a82dd72d9b\") " pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" Oct 11 10:51:57.020644 master-1 kubenswrapper[4771]: I1011 10:51:57.020277 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-pnmnt\" (UniqueName: \"kubernetes.io/projected/608a645c-104f-4fab-b1c6-cbcae70ca0f4-kube-api-access-pnmnt\") pod \"heat-operator-controller-manager-68fc865f87-dfx76\" (UID: \"608a645c-104f-4fab-b1c6-cbcae70ca0f4\") " pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" Oct 11 10:51:57.057708 master-1 kubenswrapper[4771]: I1011 10:51:57.057636 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk"] Oct 11 10:51:57.058840 master-1 kubenswrapper[4771]: I1011 10:51:57.058805 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" Oct 11 10:51:57.075117 master-1 kubenswrapper[4771]: I1011 10:51:57.075065 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnmnt\" (UniqueName: \"kubernetes.io/projected/608a645c-104f-4fab-b1c6-cbcae70ca0f4-kube-api-access-pnmnt\") pod \"heat-operator-controller-manager-68fc865f87-dfx76\" (UID: \"608a645c-104f-4fab-b1c6-cbcae70ca0f4\") " pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" Oct 11 10:51:57.079028 master-1 kubenswrapper[4771]: I1011 10:51:57.078968 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk"] Oct 11 10:51:57.122474 master-1 kubenswrapper[4771]: I1011 10:51:57.122343 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-mqpww\" (UniqueName: \"kubernetes.io/projected/9f700fad-87cd-467c-9b8a-99a82dd72d9b-kube-api-access-mqpww\") pod \"ironic-operator-controller-manager-6b498574d4-brh6p\" (UID: \"9f700fad-87cd-467c-9b8a-99a82dd72d9b\") " pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" Oct 11 10:51:57.122904 master-1 kubenswrapper[4771]: I1011 10:51:57.122605 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8tlj\" (UniqueName: \"kubernetes.io/projected/5012c546-4311-4415-bb9a-9074edfc09e2-kube-api-access-n8tlj\") pod \"mariadb-operator-controller-manager-7f4856d67b-9lktk\" (UID: \"5012c546-4311-4415-bb9a-9074edfc09e2\") " pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" Oct 11 10:51:57.159472 master-1 kubenswrapper[4771]: I1011 10:51:57.159384 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqpww\" (UniqueName: \"kubernetes.io/projected/9f700fad-87cd-467c-9b8a-99a82dd72d9b-kube-api-access-mqpww\") pod \"ironic-operator-controller-manager-6b498574d4-brh6p\" (UID: \"9f700fad-87cd-467c-9b8a-99a82dd72d9b\") " pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" Oct 11 10:51:57.168569 master-1 kubenswrapper[4771]: I1011 10:51:57.168341 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7"] Oct 11 10:51:57.170805 master-1 kubenswrapper[4771]: I1011 10:51:57.170737 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" Oct 11 10:51:57.190772 master-1 kubenswrapper[4771]: I1011 10:51:57.190465 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7"] Oct 11 10:51:57.202889 master-1 kubenswrapper[4771]: I1011 10:51:57.202814 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" Oct 11 10:51:57.223393 master-1 kubenswrapper[4771]: I1011 10:51:57.223305 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjbjk\" (UniqueName: \"kubernetes.io/projected/87e9a396-b599-4e77-ab0d-24602bde55eb-kube-api-access-qjbjk\") pod \"octavia-operator-controller-manager-f456fb6cd-wnhd7\" (UID: \"87e9a396-b599-4e77-ab0d-24602bde55eb\") " pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" Oct 11 10:51:57.223708 master-1 kubenswrapper[4771]: I1011 10:51:57.223417 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-n8tlj\" (UniqueName: \"kubernetes.io/projected/5012c546-4311-4415-bb9a-9074edfc09e2-kube-api-access-n8tlj\") pod \"mariadb-operator-controller-manager-7f4856d67b-9lktk\" (UID: \"5012c546-4311-4415-bb9a-9074edfc09e2\") " pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" Oct 11 10:51:57.301286 master-1 kubenswrapper[4771]: I1011 10:51:57.301224 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8tlj\" (UniqueName: \"kubernetes.io/projected/5012c546-4311-4415-bb9a-9074edfc09e2-kube-api-access-n8tlj\") pod \"mariadb-operator-controller-manager-7f4856d67b-9lktk\" (UID: \"5012c546-4311-4415-bb9a-9074edfc09e2\") " pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" Oct 11 10:51:57.312084 master-1 kubenswrapper[4771]: I1011 10:51:57.311622 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" Oct 11 10:51:57.324462 master-1 kubenswrapper[4771]: I1011 10:51:57.324398 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-qjbjk\" (UniqueName: \"kubernetes.io/projected/87e9a396-b599-4e77-ab0d-24602bde55eb-kube-api-access-qjbjk\") pod \"octavia-operator-controller-manager-f456fb6cd-wnhd7\" (UID: \"87e9a396-b599-4e77-ab0d-24602bde55eb\") " pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" Oct 11 10:51:57.381826 master-1 kubenswrapper[4771]: I1011 10:51:57.381655 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjbjk\" (UniqueName: \"kubernetes.io/projected/87e9a396-b599-4e77-ab0d-24602bde55eb-kube-api-access-qjbjk\") pod \"octavia-operator-controller-manager-f456fb6cd-wnhd7\" (UID: \"87e9a396-b599-4e77-ab0d-24602bde55eb\") " pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" Oct 11 10:51:57.384595 master-1 kubenswrapper[4771]: I1011 10:51:57.384123 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88"] Oct 11 10:51:57.386471 master-1 kubenswrapper[4771]: I1011 10:51:57.386429 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" Oct 11 10:51:57.402469 master-1 kubenswrapper[4771]: I1011 10:51:57.402337 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" Oct 11 10:51:57.418048 master-1 kubenswrapper[4771]: I1011 10:51:57.408042 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88"] Oct 11 10:51:57.425899 master-1 kubenswrapper[4771]: I1011 10:51:57.425847 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnnwq\" (UniqueName: \"kubernetes.io/projected/a0b63855-5ade-4060-9016-a2009f5e5b45-kube-api-access-pnnwq\") pod \"telemetry-operator-controller-manager-7585684bd7-x8n88\" (UID: \"a0b63855-5ade-4060-9016-a2009f5e5b45\") " pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" Oct 11 10:51:57.426458 master-1 kubenswrapper[4771]: I1011 10:51:57.426387 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m"] Oct 11 10:51:57.428686 master-1 kubenswrapper[4771]: I1011 10:51:57.428557 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" Oct 11 10:51:57.446095 master-1 kubenswrapper[4771]: I1011 10:51:57.446037 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m"] Oct 11 10:51:57.505219 master-1 kubenswrapper[4771]: I1011 10:51:57.505025 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" Oct 11 10:51:57.526728 master-1 kubenswrapper[4771]: I1011 10:51:57.526648 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzt2\" (UniqueName: \"kubernetes.io/projected/84f24be5-1586-471c-b099-c6c81ef56674-kube-api-access-7bzt2\") pod \"test-operator-controller-manager-565dfd7bb9-bbh7m\" (UID: \"84f24be5-1586-471c-b099-c6c81ef56674\") " pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" Oct 11 10:51:57.527171 master-1 kubenswrapper[4771]: I1011 10:51:57.526938 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-pnnwq\" (UniqueName: \"kubernetes.io/projected/a0b63855-5ade-4060-9016-a2009f5e5b45-kube-api-access-pnnwq\") pod \"telemetry-operator-controller-manager-7585684bd7-x8n88\" (UID: \"a0b63855-5ade-4060-9016-a2009f5e5b45\") " pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" Oct 11 10:51:57.549752 master-1 kubenswrapper[4771]: I1011 10:51:57.549698 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnnwq\" (UniqueName: \"kubernetes.io/projected/a0b63855-5ade-4060-9016-a2009f5e5b45-kube-api-access-pnnwq\") pod \"telemetry-operator-controller-manager-7585684bd7-x8n88\" (UID: \"a0b63855-5ade-4060-9016-a2009f5e5b45\") " pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" Oct 11 10:51:57.627450 master-1 kubenswrapper[4771]: I1011 10:51:57.627343 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzt2\" (UniqueName: \"kubernetes.io/projected/84f24be5-1586-471c-b099-c6c81ef56674-kube-api-access-7bzt2\") pod \"test-operator-controller-manager-565dfd7bb9-bbh7m\" (UID: \"84f24be5-1586-471c-b099-c6c81ef56674\") " pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" Oct 11 10:51:57.688459 master-1 kubenswrapper[4771]: I1011 10:51:57.688183 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzt2\" (UniqueName: \"kubernetes.io/projected/84f24be5-1586-471c-b099-c6c81ef56674-kube-api-access-7bzt2\") pod \"test-operator-controller-manager-565dfd7bb9-bbh7m\" (UID: \"84f24be5-1586-471c-b099-c6c81ef56674\") " pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" Oct 11 10:51:57.721597 master-1 kubenswrapper[4771]: I1011 10:51:57.721543 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" Oct 11 10:51:57.736896 master-1 kubenswrapper[4771]: I1011 10:51:57.736830 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76"] Oct 11 10:51:57.746983 master-1 kubenswrapper[4771]: I1011 10:51:57.746724 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 10:51:57.756503 master-1 kubenswrapper[4771]: I1011 10:51:57.756433 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" Oct 11 10:51:57.782019 master-1 kubenswrapper[4771]: I1011 10:51:57.781947 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp"] Oct 11 10:51:57.783162 master-1 kubenswrapper[4771]: I1011 10:51:57.783127 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp" Oct 11 10:51:57.809696 master-1 kubenswrapper[4771]: I1011 10:51:57.809416 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp"] Oct 11 10:51:57.817474 master-1 kubenswrapper[4771]: I1011 10:51:57.817427 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p"] Oct 11 10:51:57.820683 master-1 kubenswrapper[4771]: W1011 10:51:57.820504 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f700fad_87cd_467c_9b8a_99a82dd72d9b.slice/crio-112980b751234ca8e7f2218ebf4a72c9699c90415731d0169a0762eabd82689e WatchSource:0}: Error finding container 112980b751234ca8e7f2218ebf4a72c9699c90415731d0169a0762eabd82689e: Status 404 returned error can't find the container with id 112980b751234ca8e7f2218ebf4a72c9699c90415731d0169a0762eabd82689e Oct 11 10:51:57.874744 master-1 kubenswrapper[4771]: I1011 10:51:57.872547 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk"] Oct 11 10:51:57.878926 master-1 kubenswrapper[4771]: W1011 10:51:57.878675 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5012c546_4311_4415_bb9a_9074edfc09e2.slice/crio-50641270ba024a3ae8b506aa736cce25288b44114f86c69b7293c2779adc9e77 WatchSource:0}: Error finding container 50641270ba024a3ae8b506aa736cce25288b44114f86c69b7293c2779adc9e77: Status 404 returned error can't find the container with id 50641270ba024a3ae8b506aa736cce25288b44114f86c69b7293c2779adc9e77 Oct 11 10:51:57.932655 master-1 kubenswrapper[4771]: I1011 10:51:57.932434 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npkcr\" (UniqueName: \"kubernetes.io/projected/dd610cd6-c61f-4cc3-9e63-5cede4c8393b-kube-api-access-npkcr\") pod \"rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp\" (UID: \"dd610cd6-c61f-4cc3-9e63-5cede4c8393b\") " pod="openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp" Oct 11 10:51:57.993335 master-1 kubenswrapper[4771]: I1011 10:51:57.993106 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7"] Oct 11 10:51:57.993335 master-1 kubenswrapper[4771]: W1011 10:51:57.993284 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e9a396_b599_4e77_ab0d_24602bde55eb.slice/crio-57f05263c3637aefe7d23580a4f13e660b7d95dbb0bfa6de7a5815238f573e40 WatchSource:0}: Error finding container 57f05263c3637aefe7d23580a4f13e660b7d95dbb0bfa6de7a5815238f573e40: Status 404 returned error can't find the container with id 57f05263c3637aefe7d23580a4f13e660b7d95dbb0bfa6de7a5815238f573e40 Oct 11 10:51:58.035908 master-1 kubenswrapper[4771]: I1011 10:51:58.034773 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-npkcr\" (UniqueName: \"kubernetes.io/projected/dd610cd6-c61f-4cc3-9e63-5cede4c8393b-kube-api-access-npkcr\") pod \"rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp\" (UID: \"dd610cd6-c61f-4cc3-9e63-5cede4c8393b\") " pod="openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp" Oct 11 10:51:58.057570 master-1 kubenswrapper[4771]: I1011 10:51:58.057476 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-npkcr\" (UniqueName: \"kubernetes.io/projected/dd610cd6-c61f-4cc3-9e63-5cede4c8393b-kube-api-access-npkcr\") pod \"rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp\" (UID: \"dd610cd6-c61f-4cc3-9e63-5cede4c8393b\") " pod="openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp" Oct 11 10:51:58.101559 master-1 kubenswrapper[4771]: I1011 10:51:58.101478 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp" Oct 11 10:51:58.178302 master-1 kubenswrapper[4771]: I1011 10:51:58.178075 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88"] Oct 11 10:51:58.240787 master-1 kubenswrapper[4771]: I1011 10:51:58.240716 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m"] Oct 11 10:51:58.244620 master-1 kubenswrapper[4771]: W1011 10:51:58.244558 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f24be5_1586_471c_b099_c6c81ef56674.slice/crio-54b68ccfd8bee58168870a664dfb4642da4b115c8a3eb996886c35d744721edd WatchSource:0}: Error finding container 54b68ccfd8bee58168870a664dfb4642da4b115c8a3eb996886c35d744721edd: Status 404 returned error can't find the container with id 54b68ccfd8bee58168870a664dfb4642da4b115c8a3eb996886c35d744721edd Oct 11 10:51:58.315533 master-1 kubenswrapper[4771]: I1011 10:51:58.315443 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" event={"ID":"9f700fad-87cd-467c-9b8a-99a82dd72d9b","Type":"ContainerStarted","Data":"112980b751234ca8e7f2218ebf4a72c9699c90415731d0169a0762eabd82689e"} Oct 11 10:51:58.316877 master-1 kubenswrapper[4771]: I1011 10:51:58.316829 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" event={"ID":"87e9a396-b599-4e77-ab0d-24602bde55eb","Type":"ContainerStarted","Data":"57f05263c3637aefe7d23580a4f13e660b7d95dbb0bfa6de7a5815238f573e40"} Oct 11 10:51:58.317972 master-1 kubenswrapper[4771]: I1011 10:51:58.317900 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" event={"ID":"a0b63855-5ade-4060-9016-a2009f5e5b45","Type":"ContainerStarted","Data":"1417e878c4ec8c33080524292f7e4fb3af87532607bf352cb321b5d6127bf304"} Oct 11 10:51:58.319276 master-1 kubenswrapper[4771]: I1011 10:51:58.319225 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" event={"ID":"608a645c-104f-4fab-b1c6-cbcae70ca0f4","Type":"ContainerStarted","Data":"d4df31c04fb831f57ffe738508c7cd28bae0f4e0560b3d7fbc927e906485547c"} Oct 11 10:51:58.321348 master-1 kubenswrapper[4771]: I1011 10:51:58.321300 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" event={"ID":"84f24be5-1586-471c-b099-c6c81ef56674","Type":"ContainerStarted","Data":"54b68ccfd8bee58168870a664dfb4642da4b115c8a3eb996886c35d744721edd"} Oct 11 10:51:58.322492 master-1 kubenswrapper[4771]: I1011 10:51:58.322446 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" event={"ID":"5012c546-4311-4415-bb9a-9074edfc09e2","Type":"ContainerStarted","Data":"50641270ba024a3ae8b506aa736cce25288b44114f86c69b7293c2779adc9e77"} Oct 11 10:51:58.567132 master-1 kubenswrapper[4771]: I1011 10:51:58.567067 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp"] Oct 11 10:51:59.330790 master-1 kubenswrapper[4771]: I1011 10:51:59.330706 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp" event={"ID":"dd610cd6-c61f-4cc3-9e63-5cede4c8393b","Type":"ContainerStarted","Data":"0b4affbab944663a634f511679cdcb6056a1dd538220c52ad684e595b571f6f1"} Oct 11 10:52:05.367640 master-1 kubenswrapper[4771]: I1011 10:52:05.367579 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" event={"ID":"a0b63855-5ade-4060-9016-a2009f5e5b45","Type":"ContainerStarted","Data":"61b1d86db60d1ef08fae745bff4e2990d6e0710f297a7cbcad6ec61d4e26a9f8"} Oct 11 10:52:05.369153 master-1 kubenswrapper[4771]: I1011 10:52:05.369120 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" event={"ID":"84f24be5-1586-471c-b099-c6c81ef56674","Type":"ContainerStarted","Data":"4bbd4bd58f0a4e2f72e37fe8d911d861215c2e73d1ac55e10c3ef1408f3aef45"} Oct 11 10:52:05.370493 master-1 kubenswrapper[4771]: I1011 10:52:05.370449 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" event={"ID":"5012c546-4311-4415-bb9a-9074edfc09e2","Type":"ContainerStarted","Data":"ee5b794b3c7eb6a1579647ac5e38c47fb4a4c7cdbae1bb87c486b4cc4f5b9223"} Oct 11 10:52:05.372507 master-1 kubenswrapper[4771]: I1011 10:52:05.372469 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp" event={"ID":"dd610cd6-c61f-4cc3-9e63-5cede4c8393b","Type":"ContainerStarted","Data":"33f5861033987a587a008dbb0dfec7e74f5f027174580864f8bffd1335bba07d"} Oct 11 10:52:05.374960 master-1 kubenswrapper[4771]: I1011 10:52:05.374929 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" event={"ID":"9f700fad-87cd-467c-9b8a-99a82dd72d9b","Type":"ContainerStarted","Data":"8574216a044b5ecae7dc3816f4b0af0ba4a401027097aa3dcccd6fd0a4b60348"} Oct 11 10:52:05.377909 master-1 kubenswrapper[4771]: I1011 10:52:05.377868 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" event={"ID":"87e9a396-b599-4e77-ab0d-24602bde55eb","Type":"ContainerStarted","Data":"5897aaab4a679e641ec47cdeb063815b7c319c78872e68173797b0678fcc29c2"} Oct 11 10:52:05.379983 master-1 kubenswrapper[4771]: I1011 10:52:05.379912 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" event={"ID":"608a645c-104f-4fab-b1c6-cbcae70ca0f4","Type":"ContainerStarted","Data":"202127bd84ada1b6018b4e1980a9295c708b4e02d5408de2dd43b3f8367a1267"} Oct 11 10:52:08.430836 master-1 kubenswrapper[4771]: I1011 10:52:08.430557 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" event={"ID":"a0b63855-5ade-4060-9016-a2009f5e5b45","Type":"ContainerStarted","Data":"39ff2940e88ad20d8e65ad517cceba3c07a107366731bc736d90f05ef049f0a2"} Oct 11 10:52:08.430836 master-1 kubenswrapper[4771]: I1011 10:52:08.430760 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" Oct 11 10:52:08.435103 master-1 kubenswrapper[4771]: I1011 10:52:08.435038 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" event={"ID":"608a645c-104f-4fab-b1c6-cbcae70ca0f4","Type":"ContainerStarted","Data":"54fb253aef1e7a77788b473b56432d3f6e38becd9b818ea0065dbf1c3ec23750"} Oct 11 10:52:08.435291 master-1 kubenswrapper[4771]: I1011 10:52:08.435145 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" Oct 11 10:52:08.447666 master-1 kubenswrapper[4771]: I1011 10:52:08.447633 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" Oct 11 10:52:08.447846 master-1 kubenswrapper[4771]: I1011 10:52:08.447830 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" Oct 11 10:52:08.447959 master-1 kubenswrapper[4771]: I1011 10:52:08.447945 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" Oct 11 10:52:08.448069 master-1 kubenswrapper[4771]: I1011 10:52:08.448055 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" Oct 11 10:52:08.448173 master-1 kubenswrapper[4771]: I1011 10:52:08.448150 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" event={"ID":"84f24be5-1586-471c-b099-c6c81ef56674","Type":"ContainerStarted","Data":"679d673a381d791739ef1d791993125259f18a7be676cee2655b9a9298478edc"} Oct 11 10:52:08.448260 master-1 kubenswrapper[4771]: I1011 10:52:08.448242 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" event={"ID":"5012c546-4311-4415-bb9a-9074edfc09e2","Type":"ContainerStarted","Data":"2582341f8c26803c35a24f8876b1925d84cf18e54e5a19da5835825ff331e8d8"} Oct 11 10:52:08.448378 master-1 kubenswrapper[4771]: I1011 10:52:08.448338 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" event={"ID":"9f700fad-87cd-467c-9b8a-99a82dd72d9b","Type":"ContainerStarted","Data":"14fb7255e1575650c0260cc1228253177bc2593ddaa3b312e2c4a4712d460003"} Oct 11 10:52:08.448499 master-1 kubenswrapper[4771]: I1011 10:52:08.448478 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" event={"ID":"87e9a396-b599-4e77-ab0d-24602bde55eb","Type":"ContainerStarted","Data":"0bb362d02625f8307ac99882c21bf4c6cd85b865a50f7365d799ea9a8117753f"} Oct 11 10:52:08.959959 master-1 kubenswrapper[4771]: I1011 10:52:08.959858 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack-operators/rabbitmq-cluster-operator-manager-84795b7cfd-zrnpp" podStartSLOduration=6.045027368 podStartE2EDuration="11.959829506s" podCreationTimestamp="2025-10-11 10:51:57 +0000 UTC" firstStartedPulling="2025-10-11 10:51:58.573883421 +0000 UTC m=+1550.548109902" lastFinishedPulling="2025-10-11 10:52:04.488685599 +0000 UTC m=+1556.462912040" observedRunningTime="2025-10-11 10:52:05.398984267 +0000 UTC m=+1557.373210728" watchObservedRunningTime="2025-10-11 10:52:08.959829506 +0000 UTC m=+1560.934055987" Oct 11 10:52:08.960955 master-1 kubenswrapper[4771]: I1011 10:52:08.960885 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" podStartSLOduration=2.906456855 podStartE2EDuration="11.960872077s" podCreationTimestamp="2025-10-11 10:51:57 +0000 UTC" firstStartedPulling="2025-10-11 10:51:58.192769278 +0000 UTC m=+1550.166995719" lastFinishedPulling="2025-10-11 10:52:07.2471845 +0000 UTC m=+1559.221410941" observedRunningTime="2025-10-11 10:52:08.804564828 +0000 UTC m=+1560.778791339" watchObservedRunningTime="2025-10-11 10:52:08.960872077 +0000 UTC m=+1560.935098548" Oct 11 10:52:09.070115 master-1 kubenswrapper[4771]: I1011 10:52:09.069304 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" podStartSLOduration=3.729008553 podStartE2EDuration="13.069262177s" podCreationTimestamp="2025-10-11 10:51:56 +0000 UTC" firstStartedPulling="2025-10-11 10:51:57.825598639 +0000 UTC m=+1549.799825080" lastFinishedPulling="2025-10-11 10:52:07.165852273 +0000 UTC m=+1559.140078704" observedRunningTime="2025-10-11 10:52:09.055750116 +0000 UTC m=+1561.029976637" watchObservedRunningTime="2025-10-11 10:52:09.069262177 +0000 UTC m=+1561.043488678" Oct 11 10:52:09.084229 master-1 kubenswrapper[4771]: I1011 10:52:09.084128 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" podStartSLOduration=3.372610996 podStartE2EDuration="13.084104228s" podCreationTimestamp="2025-10-11 10:51:56 +0000 UTC" firstStartedPulling="2025-10-11 10:51:57.746463956 +0000 UTC m=+1549.720690397" lastFinishedPulling="2025-10-11 10:52:07.457957188 +0000 UTC m=+1559.432183629" observedRunningTime="2025-10-11 10:52:09.082576473 +0000 UTC m=+1561.056802934" watchObservedRunningTime="2025-10-11 10:52:09.084104228 +0000 UTC m=+1561.058330679" Oct 11 10:52:09.150171 master-1 kubenswrapper[4771]: I1011 10:52:09.150084 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" podStartSLOduration=3.051049794 podStartE2EDuration="12.150063659s" podCreationTimestamp="2025-10-11 10:51:57 +0000 UTC" firstStartedPulling="2025-10-11 10:51:58.246645659 +0000 UTC m=+1550.220872120" lastFinishedPulling="2025-10-11 10:52:07.345659544 +0000 UTC m=+1559.319885985" observedRunningTime="2025-10-11 10:52:09.147671889 +0000 UTC m=+1561.121898330" watchObservedRunningTime="2025-10-11 10:52:09.150063659 +0000 UTC m=+1561.124290110" Oct 11 10:52:09.190474 master-1 kubenswrapper[4771]: I1011 10:52:09.188517 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" podStartSLOduration=3.00947081 podStartE2EDuration="12.188482032s" podCreationTimestamp="2025-10-11 10:51:57 +0000 UTC" firstStartedPulling="2025-10-11 10:51:57.995629536 +0000 UTC m=+1549.969855977" lastFinishedPulling="2025-10-11 10:52:07.174640748 +0000 UTC m=+1559.148867199" observedRunningTime="2025-10-11 10:52:09.181775508 +0000 UTC m=+1561.156002029" watchObservedRunningTime="2025-10-11 10:52:09.188482032 +0000 UTC m=+1561.162708523" Oct 11 10:52:09.214385 master-1 kubenswrapper[4771]: I1011 10:52:09.214129 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" podStartSLOduration=2.803604225 podStartE2EDuration="12.214109485s" podCreationTimestamp="2025-10-11 10:51:57 +0000 UTC" firstStartedPulling="2025-10-11 10:51:57.881167079 +0000 UTC m=+1549.855393520" lastFinishedPulling="2025-10-11 10:52:07.291672339 +0000 UTC m=+1559.265898780" observedRunningTime="2025-10-11 10:52:09.212978652 +0000 UTC m=+1561.187205203" watchObservedRunningTime="2025-10-11 10:52:09.214109485 +0000 UTC m=+1561.188335926" Oct 11 10:52:09.458925 master-1 kubenswrapper[4771]: I1011 10:52:09.458816 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack-operators/ironic-operator-controller-manager-6b498574d4-brh6p" Oct 11 10:52:09.459747 master-1 kubenswrapper[4771]: I1011 10:52:09.459277 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack-operators/octavia-operator-controller-manager-f456fb6cd-wnhd7" Oct 11 10:52:09.459747 master-1 kubenswrapper[4771]: I1011 10:52:09.459388 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack-operators/heat-operator-controller-manager-68fc865f87-dfx76" Oct 11 10:52:09.460728 master-1 kubenswrapper[4771]: I1011 10:52:09.460693 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack-operators/mariadb-operator-controller-manager-7f4856d67b-9lktk" Oct 11 10:52:09.462839 master-1 kubenswrapper[4771]: I1011 10:52:09.462777 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack-operators/test-operator-controller-manager-565dfd7bb9-bbh7m" Oct 11 10:52:09.462911 master-1 kubenswrapper[4771]: I1011 10:52:09.462883 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack-operators/telemetry-operator-controller-manager-7585684bd7-x8n88" Oct 11 10:52:48.097977 master-1 kubenswrapper[4771]: I1011 10:52:48.097869 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-5fd846fcd9-m5nng"] Oct 11 10:52:48.099413 master-1 kubenswrapper[4771]: I1011 10:52:48.099378 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:52:48.102704 master-1 kubenswrapper[4771]: I1011 10:52:48.102658 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"dns" Oct 11 10:52:48.102911 master-1 kubenswrapper[4771]: I1011 10:52:48.102659 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"kube-root-ca.crt" Oct 11 10:52:48.102911 master-1 kubenswrapper[4771]: I1011 10:52:48.102874 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openshift-service-ca.crt" Oct 11 10:52:48.134662 master-1 kubenswrapper[4771]: I1011 10:52:48.134603 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-5fd846fcd9-m5nng"] Oct 11 10:52:48.215523 master-1 kubenswrapper[4771]: I1011 10:52:48.215477 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7be8543e-04fe-4e9d-91c9-8219b843b991-config\") pod \"dnsmasq-dns-5fd846fcd9-m5nng\" (UID: \"7be8543e-04fe-4e9d-91c9-8219b843b991\") " pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:52:48.215760 master-1 kubenswrapper[4771]: I1011 10:52:48.215538 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpgxh\" (UniqueName: \"kubernetes.io/projected/7be8543e-04fe-4e9d-91c9-8219b843b991-kube-api-access-cpgxh\") pod \"dnsmasq-dns-5fd846fcd9-m5nng\" (UID: \"7be8543e-04fe-4e9d-91c9-8219b843b991\") " pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:52:48.317408 master-1 kubenswrapper[4771]: I1011 10:52:48.317304 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7be8543e-04fe-4e9d-91c9-8219b843b991-config\") pod \"dnsmasq-dns-5fd846fcd9-m5nng\" (UID: \"7be8543e-04fe-4e9d-91c9-8219b843b991\") " pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:52:48.317408 master-1 kubenswrapper[4771]: I1011 10:52:48.317379 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-cpgxh\" (UniqueName: \"kubernetes.io/projected/7be8543e-04fe-4e9d-91c9-8219b843b991-kube-api-access-cpgxh\") pod \"dnsmasq-dns-5fd846fcd9-m5nng\" (UID: \"7be8543e-04fe-4e9d-91c9-8219b843b991\") " pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:52:48.318424 master-1 kubenswrapper[4771]: I1011 10:52:48.318389 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7be8543e-04fe-4e9d-91c9-8219b843b991-config\") pod \"dnsmasq-dns-5fd846fcd9-m5nng\" (UID: \"7be8543e-04fe-4e9d-91c9-8219b843b991\") " pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:52:48.337906 master-1 kubenswrapper[4771]: I1011 10:52:48.337820 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpgxh\" (UniqueName: \"kubernetes.io/projected/7be8543e-04fe-4e9d-91c9-8219b843b991-kube-api-access-cpgxh\") pod \"dnsmasq-dns-5fd846fcd9-m5nng\" (UID: \"7be8543e-04fe-4e9d-91c9-8219b843b991\") " pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:52:48.436643 master-1 kubenswrapper[4771]: I1011 10:52:48.436482 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:52:48.915447 master-1 kubenswrapper[4771]: I1011 10:52:48.915175 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-5fd846fcd9-m5nng"] Oct 11 10:52:48.918136 master-1 kubenswrapper[4771]: W1011 10:52:48.918047 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7be8543e_04fe_4e9d_91c9_8219b843b991.slice/crio-05a1b34b2d5123ecb163a3c60cc5758f73fbbac730006935705b7b57187290ed WatchSource:0}: Error finding container 05a1b34b2d5123ecb163a3c60cc5758f73fbbac730006935705b7b57187290ed: Status 404 returned error can't find the container with id 05a1b34b2d5123ecb163a3c60cc5758f73fbbac730006935705b7b57187290ed Oct 11 10:52:49.778018 master-1 kubenswrapper[4771]: I1011 10:52:49.777930 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" event={"ID":"7be8543e-04fe-4e9d-91c9-8219b843b991","Type":"ContainerStarted","Data":"05a1b34b2d5123ecb163a3c60cc5758f73fbbac730006935705b7b57187290ed"} Oct 11 10:52:50.530557 master-1 kubenswrapper[4771]: I1011 10:52:50.530507 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-5fd846fcd9-m5nng"] Oct 11 10:52:50.569171 master-1 kubenswrapper[4771]: I1011 10:52:50.569085 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-5b4bcc4d85-hc9b5"] Oct 11 10:52:50.571448 master-1 kubenswrapper[4771]: I1011 10:52:50.571114 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:50.575620 master-1 kubenswrapper[4771]: I1011 10:52:50.575572 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"dns-svc" Oct 11 10:52:50.582820 master-1 kubenswrapper[4771]: I1011 10:52:50.582321 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-5b4bcc4d85-hc9b5"] Oct 11 10:52:50.763938 master-1 kubenswrapper[4771]: I1011 10:52:50.763670 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r45c\" (UniqueName: \"kubernetes.io/projected/37015f12-0983-4016-9f76-6d0e3f641f28-kube-api-access-8r45c\") pod \"dnsmasq-dns-5b4bcc4d85-hc9b5\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:50.763938 master-1 kubenswrapper[4771]: I1011 10:52:50.763814 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-dns-svc\") pod \"dnsmasq-dns-5b4bcc4d85-hc9b5\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:50.763938 master-1 kubenswrapper[4771]: I1011 10:52:50.763858 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-config\") pod \"dnsmasq-dns-5b4bcc4d85-hc9b5\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:50.865171 master-1 kubenswrapper[4771]: I1011 10:52:50.865083 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-config\") pod \"dnsmasq-dns-5b4bcc4d85-hc9b5\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:50.866777 master-1 kubenswrapper[4771]: I1011 10:52:50.865190 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-8r45c\" (UniqueName: \"kubernetes.io/projected/37015f12-0983-4016-9f76-6d0e3f641f28-kube-api-access-8r45c\") pod \"dnsmasq-dns-5b4bcc4d85-hc9b5\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:50.866777 master-1 kubenswrapper[4771]: I1011 10:52:50.865242 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-dns-svc\") pod \"dnsmasq-dns-5b4bcc4d85-hc9b5\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:50.866777 master-1 kubenswrapper[4771]: I1011 10:52:50.866151 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-dns-svc\") pod \"dnsmasq-dns-5b4bcc4d85-hc9b5\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:50.868524 master-1 kubenswrapper[4771]: I1011 10:52:50.868472 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-config\") pod \"dnsmasq-dns-5b4bcc4d85-hc9b5\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:50.898524 master-1 kubenswrapper[4771]: I1011 10:52:50.898418 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r45c\" (UniqueName: \"kubernetes.io/projected/37015f12-0983-4016-9f76-6d0e3f641f28-kube-api-access-8r45c\") pod \"dnsmasq-dns-5b4bcc4d85-hc9b5\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:51.196991 master-1 kubenswrapper[4771]: I1011 10:52:51.196854 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:52:54.858009 master-1 kubenswrapper[4771]: I1011 10:52:54.855781 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/rabbitmq-server-0"] Oct 11 10:52:54.858009 master-1 kubenswrapper[4771]: I1011 10:52:54.857092 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/rabbitmq-server-0" Oct 11 10:52:54.860809 master-1 kubenswrapper[4771]: I1011 10:52:54.860759 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"rabbitmq-config-data" Oct 11 10:52:54.861112 master-1 kubenswrapper[4771]: I1011 10:52:54.861085 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"rabbitmq-server-conf" Oct 11 10:52:54.861456 master-1 kubenswrapper[4771]: I1011 10:52:54.861427 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"rabbitmq-plugins-conf" Oct 11 10:52:54.863005 master-1 kubenswrapper[4771]: I1011 10:52:54.862870 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"rabbitmq-default-user" Oct 11 10:52:54.863177 master-1 kubenswrapper[4771]: I1011 10:52:54.863100 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-rabbitmq-svc" Oct 11 10:52:54.863414 master-1 kubenswrapper[4771]: I1011 10:52:54.862647 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"rabbitmq-erlang-cookie" Oct 11 10:52:54.879616 master-1 kubenswrapper[4771]: I1011 10:52:54.879526 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/rabbitmq-server-0"] Oct 11 10:52:55.026731 master-1 kubenswrapper[4771]: I1011 10:52:55.026675 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"server-conf\" (UniqueName: \"kubernetes.io/configmap/6fe99fba-e358-4203-a516-04b9ae19d789-server-conf\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.026958 master-1 kubenswrapper[4771]: I1011 10:52:55.026724 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rabbitmq-tls\" (UniqueName: \"kubernetes.io/projected/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-tls\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.027006 master-1 kubenswrapper[4771]: I1011 10:52:55.026975 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-conf\" (UniqueName: \"kubernetes.io/configmap/6fe99fba-e358-4203-a516-04b9ae19d789-plugins-conf\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.027047 master-1 kubenswrapper[4771]: I1011 10:52:55.027011 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"erlang-cookie-secret\" (UniqueName: \"kubernetes.io/secret/6fe99fba-e358-4203-a516-04b9ae19d789-erlang-cookie-secret\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.027047 master-1 kubenswrapper[4771]: I1011 10:52:55.027034 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rabbitmq-erlang-cookie\" (UniqueName: \"kubernetes.io/empty-dir/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-erlang-cookie\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.027107 master-1 kubenswrapper[4771]: I1011 10:52:55.027062 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvfr5\" (UniqueName: \"kubernetes.io/projected/6fe99fba-e358-4203-a516-04b9ae19d789-kube-api-access-hvfr5\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.027107 master-1 kubenswrapper[4771]: I1011 10:52:55.027085 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rabbitmq-confd\" (UniqueName: \"kubernetes.io/projected/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-confd\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.027166 master-1 kubenswrapper[4771]: I1011 10:52:55.027116 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/configmap/6fe99fba-e358-4203-a516-04b9ae19d789-config-data\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.027166 master-1 kubenswrapper[4771]: I1011 10:52:55.027135 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rabbitmq-plugins\" (UniqueName: \"kubernetes.io/empty-dir/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-plugins\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.027166 master-1 kubenswrapper[4771]: I1011 10:52:55.027156 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pod-info\" (UniqueName: \"kubernetes.io/downward-api/6fe99fba-e358-4203-a516-04b9ae19d789-pod-info\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.027263 master-1 kubenswrapper[4771]: I1011 10:52:55.027175 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-3f889608-8928-49b4-887e-c3f52b41fe53\" (UniqueName: \"kubernetes.io/csi/topolvm.io^c0542fa6-eb99-4081-a1c8-ffbcb1c5f846\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129507 master-1 kubenswrapper[4771]: I1011 10:52:55.129273 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"plugins-conf\" (UniqueName: \"kubernetes.io/configmap/6fe99fba-e358-4203-a516-04b9ae19d789-plugins-conf\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129507 master-1 kubenswrapper[4771]: I1011 10:52:55.129474 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"erlang-cookie-secret\" (UniqueName: \"kubernetes.io/secret/6fe99fba-e358-4203-a516-04b9ae19d789-erlang-cookie-secret\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129873 master-1 kubenswrapper[4771]: I1011 10:52:55.129514 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"rabbitmq-erlang-cookie\" (UniqueName: \"kubernetes.io/empty-dir/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-erlang-cookie\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129873 master-1 kubenswrapper[4771]: I1011 10:52:55.129556 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfr5\" (UniqueName: \"kubernetes.io/projected/6fe99fba-e358-4203-a516-04b9ae19d789-kube-api-access-hvfr5\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129873 master-1 kubenswrapper[4771]: I1011 10:52:55.129591 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"rabbitmq-confd\" (UniqueName: \"kubernetes.io/projected/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-confd\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129873 master-1 kubenswrapper[4771]: I1011 10:52:55.129636 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/configmap/6fe99fba-e358-4203-a516-04b9ae19d789-config-data\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129873 master-1 kubenswrapper[4771]: I1011 10:52:55.129663 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"rabbitmq-plugins\" (UniqueName: \"kubernetes.io/empty-dir/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-plugins\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129873 master-1 kubenswrapper[4771]: I1011 10:52:55.129695 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pod-info\" (UniqueName: \"kubernetes.io/downward-api/6fe99fba-e358-4203-a516-04b9ae19d789-pod-info\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129873 master-1 kubenswrapper[4771]: I1011 10:52:55.129723 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-3f889608-8928-49b4-887e-c3f52b41fe53\" (UniqueName: \"kubernetes.io/csi/topolvm.io^c0542fa6-eb99-4081-a1c8-ffbcb1c5f846\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129873 master-1 kubenswrapper[4771]: I1011 10:52:55.129774 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"server-conf\" (UniqueName: \"kubernetes.io/configmap/6fe99fba-e358-4203-a516-04b9ae19d789-server-conf\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.129873 master-1 kubenswrapper[4771]: I1011 10:52:55.129804 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"rabbitmq-tls\" (UniqueName: \"kubernetes.io/projected/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-tls\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.131922 master-1 kubenswrapper[4771]: I1011 10:52:55.130957 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"rabbitmq-erlang-cookie\" (UniqueName: \"kubernetes.io/empty-dir/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-erlang-cookie\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.132513 master-1 kubenswrapper[4771]: I1011 10:52:55.132019 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/configmap/6fe99fba-e358-4203-a516-04b9ae19d789-config-data\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.132513 master-1 kubenswrapper[4771]: I1011 10:52:55.132254 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"rabbitmq-plugins\" (UniqueName: \"kubernetes.io/empty-dir/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-plugins\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.134240 master-1 kubenswrapper[4771]: I1011 10:52:55.133225 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"plugins-conf\" (UniqueName: \"kubernetes.io/configmap/6fe99fba-e358-4203-a516-04b9ae19d789-plugins-conf\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.134240 master-1 kubenswrapper[4771]: I1011 10:52:55.134193 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"server-conf\" (UniqueName: \"kubernetes.io/configmap/6fe99fba-e358-4203-a516-04b9ae19d789-server-conf\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.136438 master-1 kubenswrapper[4771]: I1011 10:52:55.136398 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"rabbitmq-tls\" (UniqueName: \"kubernetes.io/projected/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-tls\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.136614 master-1 kubenswrapper[4771]: I1011 10:52:55.136582 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:52:55.136669 master-1 kubenswrapper[4771]: I1011 10:52:55.136627 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-3f889608-8928-49b4-887e-c3f52b41fe53\" (UniqueName: \"kubernetes.io/csi/topolvm.io^c0542fa6-eb99-4081-a1c8-ffbcb1c5f846\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/e4a8b82c9fe18ef491f81f118a31075ad74b50c23080fed1324dd231fdb36208/globalmount\"" pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.139520 master-1 kubenswrapper[4771]: I1011 10:52:55.139478 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"rabbitmq-confd\" (UniqueName: \"kubernetes.io/projected/6fe99fba-e358-4203-a516-04b9ae19d789-rabbitmq-confd\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.139616 master-1 kubenswrapper[4771]: I1011 10:52:55.139490 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pod-info\" (UniqueName: \"kubernetes.io/downward-api/6fe99fba-e358-4203-a516-04b9ae19d789-pod-info\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.146460 master-1 kubenswrapper[4771]: I1011 10:52:55.145381 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"erlang-cookie-secret\" (UniqueName: \"kubernetes.io/secret/6fe99fba-e358-4203-a516-04b9ae19d789-erlang-cookie-secret\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.155321 master-1 kubenswrapper[4771]: I1011 10:52:55.155232 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfr5\" (UniqueName: \"kubernetes.io/projected/6fe99fba-e358-4203-a516-04b9ae19d789-kube-api-access-hvfr5\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:55.163642 master-1 kubenswrapper[4771]: I1011 10:52:55.163567 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/memcached-0"] Oct 11 10:52:55.165432 master-1 kubenswrapper[4771]: I1011 10:52:55.165348 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/memcached-0" Oct 11 10:52:55.169922 master-1 kubenswrapper[4771]: I1011 10:52:55.169671 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-memcached-svc" Oct 11 10:52:55.171428 master-1 kubenswrapper[4771]: I1011 10:52:55.171067 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"memcached-config-data" Oct 11 10:52:55.181598 master-1 kubenswrapper[4771]: I1011 10:52:55.181535 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"combined-ca-bundle" Oct 11 10:52:55.195218 master-1 kubenswrapper[4771]: I1011 10:52:55.195067 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/memcached-0"] Oct 11 10:52:55.231247 master-1 kubenswrapper[4771]: I1011 10:52:55.231178 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kolla-config\" (UniqueName: \"kubernetes.io/configmap/3843977f-b9b9-4f98-9205-5dbe3113fa5e-kolla-config\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.235427 master-1 kubenswrapper[4771]: I1011 10:52:55.235388 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3843977f-b9b9-4f98-9205-5dbe3113fa5e-combined-ca-bundle\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.235647 master-1 kubenswrapper[4771]: I1011 10:52:55.235621 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"memcached-tls-certs\" (UniqueName: \"kubernetes.io/secret/3843977f-b9b9-4f98-9205-5dbe3113fa5e-memcached-tls-certs\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.235840 master-1 kubenswrapper[4771]: I1011 10:52:55.235820 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdhgr\" (UniqueName: \"kubernetes.io/projected/3843977f-b9b9-4f98-9205-5dbe3113fa5e-kube-api-access-xdhgr\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.235989 master-1 kubenswrapper[4771]: I1011 10:52:55.235969 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/configmap/3843977f-b9b9-4f98-9205-5dbe3113fa5e-config-data\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.337897 master-1 kubenswrapper[4771]: I1011 10:52:55.337811 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kolla-config\" (UniqueName: \"kubernetes.io/configmap/3843977f-b9b9-4f98-9205-5dbe3113fa5e-kolla-config\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.338218 master-1 kubenswrapper[4771]: I1011 10:52:55.337964 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3843977f-b9b9-4f98-9205-5dbe3113fa5e-combined-ca-bundle\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.338218 master-1 kubenswrapper[4771]: I1011 10:52:55.338004 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"memcached-tls-certs\" (UniqueName: \"kubernetes.io/secret/3843977f-b9b9-4f98-9205-5dbe3113fa5e-memcached-tls-certs\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.338218 master-1 kubenswrapper[4771]: I1011 10:52:55.338055 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-xdhgr\" (UniqueName: \"kubernetes.io/projected/3843977f-b9b9-4f98-9205-5dbe3113fa5e-kube-api-access-xdhgr\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.338218 master-1 kubenswrapper[4771]: I1011 10:52:55.338104 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/configmap/3843977f-b9b9-4f98-9205-5dbe3113fa5e-config-data\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.339389 master-1 kubenswrapper[4771]: I1011 10:52:55.339333 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kolla-config\" (UniqueName: \"kubernetes.io/configmap/3843977f-b9b9-4f98-9205-5dbe3113fa5e-kolla-config\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.339511 master-1 kubenswrapper[4771]: I1011 10:52:55.339474 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/configmap/3843977f-b9b9-4f98-9205-5dbe3113fa5e-config-data\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.341203 master-1 kubenswrapper[4771]: I1011 10:52:55.341156 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3843977f-b9b9-4f98-9205-5dbe3113fa5e-combined-ca-bundle\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.342850 master-1 kubenswrapper[4771]: I1011 10:52:55.342812 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"memcached-tls-certs\" (UniqueName: \"kubernetes.io/secret/3843977f-b9b9-4f98-9205-5dbe3113fa5e-memcached-tls-certs\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.363290 master-1 kubenswrapper[4771]: I1011 10:52:55.363237 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdhgr\" (UniqueName: \"kubernetes.io/projected/3843977f-b9b9-4f98-9205-5dbe3113fa5e-kube-api-access-xdhgr\") pod \"memcached-0\" (UID: \"3843977f-b9b9-4f98-9205-5dbe3113fa5e\") " pod="openstack/memcached-0" Oct 11 10:52:55.526001 master-1 kubenswrapper[4771]: I1011 10:52:55.525795 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/memcached-0" Oct 11 10:52:56.746420 master-1 kubenswrapper[4771]: I1011 10:52:56.746324 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-3f889608-8928-49b4-887e-c3f52b41fe53\" (UniqueName: \"kubernetes.io/csi/topolvm.io^c0542fa6-eb99-4081-a1c8-ffbcb1c5f846\") pod \"rabbitmq-server-0\" (UID: \"6fe99fba-e358-4203-a516-04b9ae19d789\") " pod="openstack/rabbitmq-server-0" Oct 11 10:52:57.008117 master-1 kubenswrapper[4771]: I1011 10:52:57.007889 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/rabbitmq-server-0" Oct 11 10:53:01.117446 master-1 kubenswrapper[4771]: I1011 10:53:01.116496 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovn-controller-mtzk7"] Oct 11 10:53:01.120969 master-1 kubenswrapper[4771]: I1011 10:53:01.120884 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.126243 master-1 kubenswrapper[4771]: I1011 10:53:01.126187 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovncontroller-scripts" Oct 11 10:53:01.126776 master-1 kubenswrapper[4771]: I1011 10:53:01.126713 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ovncontroller-ovndbs" Oct 11 10:53:01.127050 master-1 kubenswrapper[4771]: I1011 10:53:01.126964 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovn-controller-ovs-mvzxp"] Oct 11 10:53:01.131341 master-1 kubenswrapper[4771]: I1011 10:53:01.131277 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.140161 master-1 kubenswrapper[4771]: I1011 10:53:01.140020 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-mtzk7"] Oct 11 10:53:01.147518 master-1 kubenswrapper[4771]: I1011 10:53:01.147460 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-ovs-mvzxp"] Oct 11 10:53:01.238797 master-1 kubenswrapper[4771]: I1011 10:53:01.232741 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/memcached-0"] Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.252554 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ldr\" (UniqueName: \"kubernetes.io/projected/71b1c323-2ebf-4a37-9327-840d3f04eda1-kube-api-access-v5ldr\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.252613 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-log\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-var-log\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.252638 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-var-run-ovn\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.252737 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-var-run\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.252804 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4jzl\" (UniqueName: \"kubernetes.io/projected/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-kube-api-access-p4jzl\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.252854 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-combined-ca-bundle\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.252935 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-var-log-ovn\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.252978 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-scripts\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.253006 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-ovs\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-etc-ovs\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.253175 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-controller-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-ovn-controller-tls-certs\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.253250 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-var-lib\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.253407 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-var-run\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.254466 master-1 kubenswrapper[4771]: I1011 10:53:01.253458 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/71b1c323-2ebf-4a37-9327-840d3f04eda1-scripts\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.282061 master-1 kubenswrapper[4771]: I1011 10:53:01.281997 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-5b4bcc4d85-hc9b5"] Oct 11 10:53:01.286270 master-1 kubenswrapper[4771]: W1011 10:53:01.286233 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37015f12_0983_4016_9f76_6d0e3f641f28.slice/crio-9218883a276b672e1ae5ba8960bf79ca7ce273871d0ff647d43beb2d404896a3 WatchSource:0}: Error finding container 9218883a276b672e1ae5ba8960bf79ca7ce273871d0ff647d43beb2d404896a3: Status 404 returned error can't find the container with id 9218883a276b672e1ae5ba8960bf79ca7ce273871d0ff647d43beb2d404896a3 Oct 11 10:53:01.357119 master-1 kubenswrapper[4771]: I1011 10:53:01.357065 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ldr\" (UniqueName: \"kubernetes.io/projected/71b1c323-2ebf-4a37-9327-840d3f04eda1-kube-api-access-v5ldr\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.357119 master-1 kubenswrapper[4771]: I1011 10:53:01.357126 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-log\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-var-log\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.357332 master-1 kubenswrapper[4771]: I1011 10:53:01.357149 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-var-run-ovn\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.357332 master-1 kubenswrapper[4771]: I1011 10:53:01.357172 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-var-run\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.357332 master-1 kubenswrapper[4771]: I1011 10:53:01.357194 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-p4jzl\" (UniqueName: \"kubernetes.io/projected/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-kube-api-access-p4jzl\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.357332 master-1 kubenswrapper[4771]: I1011 10:53:01.357216 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-combined-ca-bundle\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.357332 master-1 kubenswrapper[4771]: I1011 10:53:01.357239 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-var-log-ovn\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.357332 master-1 kubenswrapper[4771]: I1011 10:53:01.357267 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-scripts\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.357332 master-1 kubenswrapper[4771]: I1011 10:53:01.357284 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-ovs\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-etc-ovs\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.357332 master-1 kubenswrapper[4771]: I1011 10:53:01.357316 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovn-controller-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-ovn-controller-tls-certs\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.357617 master-1 kubenswrapper[4771]: I1011 10:53:01.357451 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-lib\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-var-lib\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.357617 master-1 kubenswrapper[4771]: I1011 10:53:01.357489 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-var-run\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.357617 master-1 kubenswrapper[4771]: I1011 10:53:01.357512 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/71b1c323-2ebf-4a37-9327-840d3f04eda1-scripts\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.359095 master-1 kubenswrapper[4771]: I1011 10:53:01.359068 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-log\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-var-log\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.359268 master-1 kubenswrapper[4771]: I1011 10:53:01.359215 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-var-run-ovn\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.359627 master-1 kubenswrapper[4771]: I1011 10:53:01.359584 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-ovs\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-etc-ovs\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.359676 master-1 kubenswrapper[4771]: I1011 10:53:01.359640 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-var-run\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.359729 master-1 kubenswrapper[4771]: I1011 10:53:01.359641 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-lib\" (UniqueName: \"kubernetes.io/host-path/71b1c323-2ebf-4a37-9327-840d3f04eda1-var-lib\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.359729 master-1 kubenswrapper[4771]: I1011 10:53:01.359635 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-var-run\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.359888 master-1 kubenswrapper[4771]: I1011 10:53:01.359865 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-var-log-ovn\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.361294 master-1 kubenswrapper[4771]: I1011 10:53:01.361248 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/71b1c323-2ebf-4a37-9327-840d3f04eda1-scripts\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.361440 master-1 kubenswrapper[4771]: I1011 10:53:01.361413 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-scripts\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.363133 master-1 kubenswrapper[4771]: I1011 10:53:01.363074 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovn-controller-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-ovn-controller-tls-certs\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.364543 master-1 kubenswrapper[4771]: I1011 10:53:01.364513 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-combined-ca-bundle\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.382077 master-1 kubenswrapper[4771]: I1011 10:53:01.381041 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4jzl\" (UniqueName: \"kubernetes.io/projected/4ab25521-7fba-40c9-b3db-377b1d0ec7a1-kube-api-access-p4jzl\") pod \"ovn-controller-mtzk7\" (UID: \"4ab25521-7fba-40c9-b3db-377b1d0ec7a1\") " pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.382968 master-1 kubenswrapper[4771]: I1011 10:53:01.382920 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ldr\" (UniqueName: \"kubernetes.io/projected/71b1c323-2ebf-4a37-9327-840d3f04eda1-kube-api-access-v5ldr\") pod \"ovn-controller-ovs-mvzxp\" (UID: \"71b1c323-2ebf-4a37-9327-840d3f04eda1\") " pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.388827 master-1 kubenswrapper[4771]: I1011 10:53:01.388788 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/rabbitmq-server-0"] Oct 11 10:53:01.397147 master-1 kubenswrapper[4771]: W1011 10:53:01.397083 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe99fba_e358_4203_a516_04b9ae19d789.slice/crio-338665f1b49aeaee26ab96a1458b85f97acd4442412f04354b46796f50305999 WatchSource:0}: Error finding container 338665f1b49aeaee26ab96a1458b85f97acd4442412f04354b46796f50305999: Status 404 returned error can't find the container with id 338665f1b49aeaee26ab96a1458b85f97acd4442412f04354b46796f50305999 Oct 11 10:53:01.453497 master-1 kubenswrapper[4771]: I1011 10:53:01.453425 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:01.478276 master-1 kubenswrapper[4771]: I1011 10:53:01.478222 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:01.882392 master-1 kubenswrapper[4771]: I1011 10:53:01.882318 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/rabbitmq-server-0" event={"ID":"6fe99fba-e358-4203-a516-04b9ae19d789","Type":"ContainerStarted","Data":"338665f1b49aeaee26ab96a1458b85f97acd4442412f04354b46796f50305999"} Oct 11 10:53:01.883667 master-1 kubenswrapper[4771]: I1011 10:53:01.883625 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/memcached-0" event={"ID":"3843977f-b9b9-4f98-9205-5dbe3113fa5e","Type":"ContainerStarted","Data":"2b6e1835ec110d7e4a6c5ea6f76cf1b56d9f129352c900bbd1f9ec66b48b8068"} Oct 11 10:53:01.885662 master-1 kubenswrapper[4771]: I1011 10:53:01.885588 4771 generic.go:334] "Generic (PLEG): container finished" podID="37015f12-0983-4016-9f76-6d0e3f641f28" containerID="6500f337e1810213b3c48514bdf7915497fff03dfebcbb66402d535cebd46613" exitCode=0 Oct 11 10:53:01.885662 master-1 kubenswrapper[4771]: I1011 10:53:01.885649 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" event={"ID":"37015f12-0983-4016-9f76-6d0e3f641f28","Type":"ContainerDied","Data":"6500f337e1810213b3c48514bdf7915497fff03dfebcbb66402d535cebd46613"} Oct 11 10:53:01.885821 master-1 kubenswrapper[4771]: I1011 10:53:01.885669 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" event={"ID":"37015f12-0983-4016-9f76-6d0e3f641f28","Type":"ContainerStarted","Data":"9218883a276b672e1ae5ba8960bf79ca7ce273871d0ff647d43beb2d404896a3"} Oct 11 10:53:01.888479 master-1 kubenswrapper[4771]: I1011 10:53:01.888429 4771 generic.go:334] "Generic (PLEG): container finished" podID="7be8543e-04fe-4e9d-91c9-8219b843b991" containerID="d370c75dd26b77e221c3a247825030d6ec6b2e51f63eff5e38438108d29ede13" exitCode=0 Oct 11 10:53:01.888585 master-1 kubenswrapper[4771]: I1011 10:53:01.888488 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" event={"ID":"7be8543e-04fe-4e9d-91c9-8219b843b991","Type":"ContainerDied","Data":"d370c75dd26b77e221c3a247825030d6ec6b2e51f63eff5e38438108d29ede13"} Oct 11 10:53:02.054644 master-1 kubenswrapper[4771]: I1011 10:53:02.054557 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-mtzk7"] Oct 11 10:53:02.070103 master-1 kubenswrapper[4771]: W1011 10:53:02.068226 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ab25521_7fba_40c9_b3db_377b1d0ec7a1.slice/crio-0c36d3f1d42190760f1c7787f28e6a160afd93a10732a347576180547afc6468 WatchSource:0}: Error finding container 0c36d3f1d42190760f1c7787f28e6a160afd93a10732a347576180547afc6468: Status 404 returned error can't find the container with id 0c36d3f1d42190760f1c7787f28e6a160afd93a10732a347576180547afc6468 Oct 11 10:53:02.152376 master-1 kubenswrapper[4771]: E1011 10:53:02.143529 4771 log.go:32] "CreateContainer in sandbox from runtime service failed" err=< Oct 11 10:53:02.152376 master-1 kubenswrapper[4771]: rpc error: code = Unknown desc = container create failed: mount `/var/lib/kubelet/pods/37015f12-0983-4016-9f76-6d0e3f641f28/volume-subpaths/dns-svc/dnsmasq-dns/1` to `etc/dnsmasq.d/hosts/dns-svc`: No such file or directory Oct 11 10:53:02.152376 master-1 kubenswrapper[4771]: > podSandboxID="9218883a276b672e1ae5ba8960bf79ca7ce273871d0ff647d43beb2d404896a3" Oct 11 10:53:02.152376 master-1 kubenswrapper[4771]: E1011 10:53:02.143767 4771 kuberuntime_manager.go:1274] "Unhandled Error" err=< Oct 11 10:53:02.152376 master-1 kubenswrapper[4771]: container &Container{Name:dnsmasq-dns,Image:quay.io/podified-antelope-centos9/openstack-neutron-server@sha256:c4e71b2158fd939dad8b8e705273493051d3023273d23b279f2699dce6db33df,Command:[/bin/bash],Args:[-c dnsmasq --interface=* --conf-dir=/etc/dnsmasq.d --hostsdir=/etc/dnsmasq.d/hosts --keep-in-foreground --log-debug --bind-interfaces --listen-address=$(POD_IP) --port 5353 --log-facility=- --no-hosts --domain-needed --no-resolv --bogus-priv --log-queries],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:CONFIG_HASH,Value:n86h5f9h86h67dh684h587h58hcbh68fh5d9h9hb7h659hc5h5c9h589h676h67ch579h668hc5h57h695h678h5f8hd8h664h698h5ffh664h54fhd8q,ValueFrom:nil,},EnvVar{Name:POD_IP,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:config,ReadOnly:true,MountPath:/etc/dnsmasq.d/config.cfg,SubPath:dns,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:dns-svc,ReadOnly:true,MountPath:/etc/dnsmasq.d/hosts/dns-svc,SubPath:dns-svc,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-8r45c,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:nil,TCPSocket:&TCPSocketAction{Port:{0 5353 },Host:,},GRPC:nil,},InitialDelaySeconds:3,TimeoutSeconds:5,PeriodSeconds:3,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:nil,TCPSocket:&TCPSocketAction{Port:{0 5353 },Host:,},GRPC:nil,},InitialDelaySeconds:5,TimeoutSeconds:5,PeriodSeconds:5,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[ALL],},Privileged:nil,SELinuxOptions:nil,RunAsUser:*1000790000,RunAsNonRoot:*true,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*false,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:&SeccompProfile{Type:RuntimeDefault,LocalhostProfile:nil,},AppArmorProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod dnsmasq-dns-5b4bcc4d85-hc9b5_openstack(37015f12-0983-4016-9f76-6d0e3f641f28): CreateContainerError: container create failed: mount `/var/lib/kubelet/pods/37015f12-0983-4016-9f76-6d0e3f641f28/volume-subpaths/dns-svc/dnsmasq-dns/1` to `etc/dnsmasq.d/hosts/dns-svc`: No such file or directory Oct 11 10:53:02.152376 master-1 kubenswrapper[4771]: > logger="UnhandledError" Oct 11 10:53:02.152376 master-1 kubenswrapper[4771]: E1011 10:53:02.150639 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dnsmasq-dns\" with CreateContainerError: \"container create failed: mount `/var/lib/kubelet/pods/37015f12-0983-4016-9f76-6d0e3f641f28/volume-subpaths/dns-svc/dnsmasq-dns/1` to `etc/dnsmasq.d/hosts/dns-svc`: No such file or directory\\n\"" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" podUID="37015f12-0983-4016-9f76-6d0e3f641f28" Oct 11 10:53:02.284535 master-1 kubenswrapper[4771]: I1011 10:53:02.284470 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:53:02.307170 master-1 kubenswrapper[4771]: I1011 10:53:02.307089 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/rabbitmq-cell1-server-0"] Oct 11 10:53:02.307540 master-1 kubenswrapper[4771]: E1011 10:53:02.307374 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="7be8543e-04fe-4e9d-91c9-8219b843b991" containerName="init" Oct 11 10:53:02.307540 master-1 kubenswrapper[4771]: I1011 10:53:02.307389 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="7be8543e-04fe-4e9d-91c9-8219b843b991" containerName="init" Oct 11 10:53:02.307540 master-1 kubenswrapper[4771]: I1011 10:53:02.307533 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="7be8543e-04fe-4e9d-91c9-8219b843b991" containerName="init" Oct 11 10:53:02.308321 master-1 kubenswrapper[4771]: I1011 10:53:02.308285 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.311684 master-1 kubenswrapper[4771]: I1011 10:53:02.310986 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"rabbitmq-cell1-erlang-cookie" Oct 11 10:53:02.311684 master-1 kubenswrapper[4771]: I1011 10:53:02.311433 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-rabbitmq-cell1-svc" Oct 11 10:53:02.312030 master-1 kubenswrapper[4771]: I1011 10:53:02.312005 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"rabbitmq-cell1-server-conf" Oct 11 10:53:02.312189 master-1 kubenswrapper[4771]: I1011 10:53:02.312157 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"rabbitmq-cell1-config-data" Oct 11 10:53:02.312325 master-1 kubenswrapper[4771]: I1011 10:53:02.312296 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"rabbitmq-cell1-plugins-conf" Oct 11 10:53:02.312564 master-1 kubenswrapper[4771]: I1011 10:53:02.312539 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"rabbitmq-cell1-default-user" Oct 11 10:53:02.332637 master-1 kubenswrapper[4771]: I1011 10:53:02.332566 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/rabbitmq-cell1-server-0"] Oct 11 10:53:02.377503 master-1 kubenswrapper[4771]: I1011 10:53:02.377431 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7be8543e-04fe-4e9d-91c9-8219b843b991-config\") pod \"7be8543e-04fe-4e9d-91c9-8219b843b991\" (UID: \"7be8543e-04fe-4e9d-91c9-8219b843b991\") " Oct 11 10:53:02.378075 master-1 kubenswrapper[4771]: I1011 10:53:02.377614 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpgxh\" (UniqueName: \"kubernetes.io/projected/7be8543e-04fe-4e9d-91c9-8219b843b991-kube-api-access-cpgxh\") pod \"7be8543e-04fe-4e9d-91c9-8219b843b991\" (UID: \"7be8543e-04fe-4e9d-91c9-8219b843b991\") " Oct 11 10:53:02.383894 master-1 kubenswrapper[4771]: I1011 10:53:02.383847 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be8543e-04fe-4e9d-91c9-8219b843b991-kube-api-access-cpgxh" (OuterVolumeSpecName: "kube-api-access-cpgxh") pod "7be8543e-04fe-4e9d-91c9-8219b843b991" (UID: "7be8543e-04fe-4e9d-91c9-8219b843b991"). InnerVolumeSpecName "kube-api-access-cpgxh". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:53:02.416464 master-1 kubenswrapper[4771]: I1011 10:53:02.416401 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be8543e-04fe-4e9d-91c9-8219b843b991-config" (OuterVolumeSpecName: "config") pod "7be8543e-04fe-4e9d-91c9-8219b843b991" (UID: "7be8543e-04fe-4e9d-91c9-8219b843b991"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.485799 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtjs\" (UniqueName: \"kubernetes.io/projected/831321b9-20ce-409b-8bdb-ec231aef5f35-kube-api-access-tmtjs\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.485851 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rabbitmq-confd\" (UniqueName: \"kubernetes.io/projected/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-confd\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.485875 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-conf\" (UniqueName: \"kubernetes.io/configmap/831321b9-20ce-409b-8bdb-ec231aef5f35-plugins-conf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.485946 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ac5ccdc8-e999-4333-8daa-1020a63a77e7\" (UniqueName: \"kubernetes.io/csi/topolvm.io^6a793077-de91-441f-80c8-5b4445c0ddaf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.485973 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pod-info\" (UniqueName: \"kubernetes.io/downward-api/831321b9-20ce-409b-8bdb-ec231aef5f35-pod-info\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.485997 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/configmap/831321b9-20ce-409b-8bdb-ec231aef5f35-config-data\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.486014 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rabbitmq-erlang-cookie\" (UniqueName: \"kubernetes.io/empty-dir/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-erlang-cookie\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.486040 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"erlang-cookie-secret\" (UniqueName: \"kubernetes.io/secret/831321b9-20ce-409b-8bdb-ec231aef5f35-erlang-cookie-secret\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.486055 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rabbitmq-plugins\" (UniqueName: \"kubernetes.io/empty-dir/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-plugins\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.486218 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rabbitmq-tls\" (UniqueName: \"kubernetes.io/projected/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-tls\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.486237 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"server-conf\" (UniqueName: \"kubernetes.io/configmap/831321b9-20ce-409b-8bdb-ec231aef5f35-server-conf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.486280 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-cpgxh\" (UniqueName: \"kubernetes.io/projected/7be8543e-04fe-4e9d-91c9-8219b843b991-kube-api-access-cpgxh\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:02.487459 master-1 kubenswrapper[4771]: I1011 10:53:02.486295 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7be8543e-04fe-4e9d-91c9-8219b843b991-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588166 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtjs\" (UniqueName: \"kubernetes.io/projected/831321b9-20ce-409b-8bdb-ec231aef5f35-kube-api-access-tmtjs\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588271 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"rabbitmq-confd\" (UniqueName: \"kubernetes.io/projected/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-confd\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588302 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"plugins-conf\" (UniqueName: \"kubernetes.io/configmap/831321b9-20ce-409b-8bdb-ec231aef5f35-plugins-conf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588339 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-ac5ccdc8-e999-4333-8daa-1020a63a77e7\" (UniqueName: \"kubernetes.io/csi/topolvm.io^6a793077-de91-441f-80c8-5b4445c0ddaf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588406 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pod-info\" (UniqueName: \"kubernetes.io/downward-api/831321b9-20ce-409b-8bdb-ec231aef5f35-pod-info\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588442 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/configmap/831321b9-20ce-409b-8bdb-ec231aef5f35-config-data\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588469 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"rabbitmq-erlang-cookie\" (UniqueName: \"kubernetes.io/empty-dir/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-erlang-cookie\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588517 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"rabbitmq-plugins\" (UniqueName: \"kubernetes.io/empty-dir/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-plugins\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588539 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"erlang-cookie-secret\" (UniqueName: \"kubernetes.io/secret/831321b9-20ce-409b-8bdb-ec231aef5f35-erlang-cookie-secret\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588606 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"rabbitmq-tls\" (UniqueName: \"kubernetes.io/projected/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-tls\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.588632 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"server-conf\" (UniqueName: \"kubernetes.io/configmap/831321b9-20ce-409b-8bdb-ec231aef5f35-server-conf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.589592 master-1 kubenswrapper[4771]: I1011 10:53:02.589155 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"rabbitmq-erlang-cookie\" (UniqueName: \"kubernetes.io/empty-dir/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-erlang-cookie\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.590693 master-1 kubenswrapper[4771]: I1011 10:53:02.589989 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"plugins-conf\" (UniqueName: \"kubernetes.io/configmap/831321b9-20ce-409b-8bdb-ec231aef5f35-plugins-conf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.590693 master-1 kubenswrapper[4771]: I1011 10:53:02.590274 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"rabbitmq-plugins\" (UniqueName: \"kubernetes.io/empty-dir/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-plugins\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.591540 master-1 kubenswrapper[4771]: I1011 10:53:02.591493 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/configmap/831321b9-20ce-409b-8bdb-ec231aef5f35-config-data\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.591934 master-1 kubenswrapper[4771]: I1011 10:53:02.591902 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:53:02.591993 master-1 kubenswrapper[4771]: I1011 10:53:02.591935 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-ac5ccdc8-e999-4333-8daa-1020a63a77e7\" (UniqueName: \"kubernetes.io/csi/topolvm.io^6a793077-de91-441f-80c8-5b4445c0ddaf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/db0f3e0ea24975b05cacd28424c426b113bc2abc30785b5ccd7afb590600ef18/globalmount\"" pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.592833 master-1 kubenswrapper[4771]: I1011 10:53:02.592790 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"rabbitmq-confd\" (UniqueName: \"kubernetes.io/projected/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-confd\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.593274 master-1 kubenswrapper[4771]: I1011 10:53:02.593239 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"server-conf\" (UniqueName: \"kubernetes.io/configmap/831321b9-20ce-409b-8bdb-ec231aef5f35-server-conf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.593274 master-1 kubenswrapper[4771]: I1011 10:53:02.593258 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pod-info\" (UniqueName: \"kubernetes.io/downward-api/831321b9-20ce-409b-8bdb-ec231aef5f35-pod-info\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.595242 master-1 kubenswrapper[4771]: I1011 10:53:02.595203 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"erlang-cookie-secret\" (UniqueName: \"kubernetes.io/secret/831321b9-20ce-409b-8bdb-ec231aef5f35-erlang-cookie-secret\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.595972 master-1 kubenswrapper[4771]: I1011 10:53:02.595936 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"rabbitmq-tls\" (UniqueName: \"kubernetes.io/projected/831321b9-20ce-409b-8bdb-ec231aef5f35-rabbitmq-tls\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.596039 master-1 kubenswrapper[4771]: I1011 10:53:02.596018 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-ovs-mvzxp"] Oct 11 10:53:02.613301 master-1 kubenswrapper[4771]: I1011 10:53:02.613248 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtjs\" (UniqueName: \"kubernetes.io/projected/831321b9-20ce-409b-8bdb-ec231aef5f35-kube-api-access-tmtjs\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:02.614432 master-1 kubenswrapper[4771]: W1011 10:53:02.614392 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71b1c323_2ebf_4a37_9327_840d3f04eda1.slice/crio-b886f32cad28d7dd63a7e66891a5ff4e56a52d75e0ab3b67c6a7dd18be269991 WatchSource:0}: Error finding container b886f32cad28d7dd63a7e66891a5ff4e56a52d75e0ab3b67c6a7dd18be269991: Status 404 returned error can't find the container with id b886f32cad28d7dd63a7e66891a5ff4e56a52d75e0ab3b67c6a7dd18be269991 Oct 11 10:53:02.892875 master-1 kubenswrapper[4771]: I1011 10:53:02.892794 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovn-controller-metrics-xvjvs"] Oct 11 10:53:02.893905 master-1 kubenswrapper[4771]: I1011 10:53:02.893879 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:02.897175 master-1 kubenswrapper[4771]: I1011 10:53:02.897121 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovncontroller-metrics-config" Oct 11 10:53:02.897421 master-1 kubenswrapper[4771]: I1011 10:53:02.897221 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ovn-metrics" Oct 11 10:53:02.905059 master-1 kubenswrapper[4771]: I1011 10:53:02.904997 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-ovs-mvzxp" event={"ID":"71b1c323-2ebf-4a37-9327-840d3f04eda1","Type":"ContainerStarted","Data":"b886f32cad28d7dd63a7e66891a5ff4e56a52d75e0ab3b67c6a7dd18be269991"} Oct 11 10:53:02.909178 master-1 kubenswrapper[4771]: I1011 10:53:02.909075 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7" event={"ID":"4ab25521-7fba-40c9-b3db-377b1d0ec7a1","Type":"ContainerStarted","Data":"0c36d3f1d42190760f1c7787f28e6a160afd93a10732a347576180547afc6468"} Oct 11 10:53:02.911994 master-1 kubenswrapper[4771]: I1011 10:53:02.911961 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-metrics-xvjvs"] Oct 11 10:53:02.912677 master-1 kubenswrapper[4771]: I1011 10:53:02.912652 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" Oct 11 10:53:02.912765 master-1 kubenswrapper[4771]: I1011 10:53:02.912689 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-5fd846fcd9-m5nng" event={"ID":"7be8543e-04fe-4e9d-91c9-8219b843b991","Type":"ContainerDied","Data":"05a1b34b2d5123ecb163a3c60cc5758f73fbbac730006935705b7b57187290ed"} Oct 11 10:53:02.912836 master-1 kubenswrapper[4771]: I1011 10:53:02.912764 4771 scope.go:117] "RemoveContainer" containerID="d370c75dd26b77e221c3a247825030d6ec6b2e51f63eff5e38438108d29ede13" Oct 11 10:53:02.982682 master-1 kubenswrapper[4771]: I1011 10:53:02.982557 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-5fd846fcd9-m5nng"] Oct 11 10:53:02.993172 master-1 kubenswrapper[4771]: I1011 10:53:02.992999 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-5fd846fcd9-m5nng"] Oct 11 10:53:02.994916 master-1 kubenswrapper[4771]: I1011 10:53:02.994890 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-metrics-certs-tls-certs\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:02.995137 master-1 kubenswrapper[4771]: I1011 10:53:02.995117 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-config\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:02.995291 master-1 kubenswrapper[4771]: I1011 10:53:02.995271 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovs-rundir\" (UniqueName: \"kubernetes.io/host-path/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-ovs-rundir\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:02.995476 master-1 kubenswrapper[4771]: I1011 10:53:02.995461 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-rundir\" (UniqueName: \"kubernetes.io/host-path/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-ovn-rundir\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:02.995593 master-1 kubenswrapper[4771]: I1011 10:53:02.995573 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-combined-ca-bundle\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:02.995740 master-1 kubenswrapper[4771]: I1011 10:53:02.995725 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rnsn\" (UniqueName: \"kubernetes.io/projected/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-kube-api-access-8rnsn\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.102927 master-1 kubenswrapper[4771]: I1011 10:53:03.097024 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-combined-ca-bundle\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.103645 master-1 kubenswrapper[4771]: I1011 10:53:03.100575 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-combined-ca-bundle\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.103837 master-1 kubenswrapper[4771]: I1011 10:53:03.103798 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-8rnsn\" (UniqueName: \"kubernetes.io/projected/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-kube-api-access-8rnsn\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.104020 master-1 kubenswrapper[4771]: I1011 10:53:03.104001 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-metrics-certs-tls-certs\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.104344 master-1 kubenswrapper[4771]: I1011 10:53:03.104325 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-config\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.104520 master-1 kubenswrapper[4771]: I1011 10:53:03.104501 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovs-rundir\" (UniqueName: \"kubernetes.io/host-path/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-ovs-rundir\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.104623 master-1 kubenswrapper[4771]: I1011 10:53:03.104594 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovs-rundir\" (UniqueName: \"kubernetes.io/host-path/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-ovs-rundir\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.104904 master-1 kubenswrapper[4771]: I1011 10:53:03.104881 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovn-rundir\" (UniqueName: \"kubernetes.io/host-path/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-ovn-rundir\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.105073 master-1 kubenswrapper[4771]: I1011 10:53:03.104948 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovn-rundir\" (UniqueName: \"kubernetes.io/host-path/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-ovn-rundir\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.105557 master-1 kubenswrapper[4771]: I1011 10:53:03.105492 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-config\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.108770 master-1 kubenswrapper[4771]: I1011 10:53:03.108312 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-metrics-certs-tls-certs\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.124485 master-1 kubenswrapper[4771]: I1011 10:53:03.124428 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rnsn\" (UniqueName: \"kubernetes.io/projected/a0617ee1-7793-4630-9daa-5d4d02f1c5fe-kube-api-access-8rnsn\") pod \"ovn-controller-metrics-xvjvs\" (UID: \"a0617ee1-7793-4630-9daa-5d4d02f1c5fe\") " pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:03.218443 master-1 kubenswrapper[4771]: I1011 10:53:03.217178 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-metrics-xvjvs" Oct 11 10:53:04.226311 master-1 kubenswrapper[4771]: I1011 10:53:04.226258 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-ac5ccdc8-e999-4333-8daa-1020a63a77e7\" (UniqueName: \"kubernetes.io/csi/topolvm.io^6a793077-de91-441f-80c8-5b4445c0ddaf\") pod \"rabbitmq-cell1-server-0\" (UID: \"831321b9-20ce-409b-8bdb-ec231aef5f35\") " pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:04.440500 master-1 kubenswrapper[4771]: I1011 10:53:04.439929 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:04.449691 master-1 kubenswrapper[4771]: I1011 10:53:04.449629 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7be8543e-04fe-4e9d-91c9-8219b843b991" path="/var/lib/kubelet/pods/7be8543e-04fe-4e9d-91c9-8219b843b991/volumes" Oct 11 10:53:05.361108 master-1 kubenswrapper[4771]: I1011 10:53:05.361008 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/openstack-galera-0"] Oct 11 10:53:05.362503 master-1 kubenswrapper[4771]: I1011 10:53:05.362476 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/openstack-galera-0" Oct 11 10:53:05.365873 master-1 kubenswrapper[4771]: I1011 10:53:05.365817 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-scripts" Oct 11 10:53:05.366284 master-1 kubenswrapper[4771]: I1011 10:53:05.366249 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"osp-secret" Oct 11 10:53:05.366641 master-1 kubenswrapper[4771]: I1011 10:53:05.366617 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-config-data" Oct 11 10:53:05.374729 master-1 kubenswrapper[4771]: I1011 10:53:05.374688 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-galera-openstack-svc" Oct 11 10:53:05.490382 master-1 kubenswrapper[4771]: I1011 10:53:05.484063 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/openstack-galera-0"] Oct 11 10:53:05.551092 master-1 kubenswrapper[4771]: I1011 10:53:05.551015 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-generated\" (UniqueName: \"kubernetes.io/empty-dir/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-config-data-generated\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.551400 master-1 kubenswrapper[4771]: I1011 10:53:05.551225 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secrets\" (UniqueName: \"kubernetes.io/secret/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-secrets\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.551400 master-1 kubenswrapper[4771]: I1011 10:53:05.551372 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-default\" (UniqueName: \"kubernetes.io/configmap/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-config-data-default\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.551477 master-1 kubenswrapper[4771]: I1011 10:53:05.551409 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-scripts\" (UniqueName: \"kubernetes.io/configmap/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-operator-scripts\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.551546 master-1 kubenswrapper[4771]: I1011 10:53:05.551522 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-combined-ca-bundle\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.551664 master-1 kubenswrapper[4771]: I1011 10:53:05.551641 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnm9t\" (UniqueName: \"kubernetes.io/projected/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-kube-api-access-fnm9t\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.551740 master-1 kubenswrapper[4771]: I1011 10:53:05.551727 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6ff4ce29-e30c-42fb-a4c8-2f59d5ba9d6a\" (UniqueName: \"kubernetes.io/csi/topolvm.io^050fd1fe-ba08-48fe-9f1b-0beca688083e\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.551826 master-1 kubenswrapper[4771]: I1011 10:53:05.551809 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kolla-config\" (UniqueName: \"kubernetes.io/configmap/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-kolla-config\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.551929 master-1 kubenswrapper[4771]: I1011 10:53:05.551843 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"galera-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-galera-tls-certs\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.654209 master-1 kubenswrapper[4771]: I1011 10:53:05.653966 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-default\" (UniqueName: \"kubernetes.io/configmap/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-config-data-default\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.654209 master-1 kubenswrapper[4771]: I1011 10:53:05.654107 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"operator-scripts\" (UniqueName: \"kubernetes.io/configmap/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-operator-scripts\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.654762 master-1 kubenswrapper[4771]: I1011 10:53:05.654180 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-combined-ca-bundle\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.654762 master-1 kubenswrapper[4771]: I1011 10:53:05.654591 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-fnm9t\" (UniqueName: \"kubernetes.io/projected/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-kube-api-access-fnm9t\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.654762 master-1 kubenswrapper[4771]: I1011 10:53:05.654692 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-6ff4ce29-e30c-42fb-a4c8-2f59d5ba9d6a\" (UniqueName: \"kubernetes.io/csi/topolvm.io^050fd1fe-ba08-48fe-9f1b-0beca688083e\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.654893 master-1 kubenswrapper[4771]: I1011 10:53:05.654774 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kolla-config\" (UniqueName: \"kubernetes.io/configmap/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-kolla-config\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.654893 master-1 kubenswrapper[4771]: I1011 10:53:05.654801 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"galera-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-galera-tls-certs\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.654893 master-1 kubenswrapper[4771]: I1011 10:53:05.654876 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-generated\" (UniqueName: \"kubernetes.io/empty-dir/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-config-data-generated\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.654990 master-1 kubenswrapper[4771]: I1011 10:53:05.654944 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secrets\" (UniqueName: \"kubernetes.io/secret/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-secrets\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.655955 master-1 kubenswrapper[4771]: I1011 10:53:05.655717 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-generated\" (UniqueName: \"kubernetes.io/empty-dir/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-config-data-generated\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.655955 master-1 kubenswrapper[4771]: I1011 10:53:05.655724 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-default\" (UniqueName: \"kubernetes.io/configmap/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-config-data-default\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.656119 master-1 kubenswrapper[4771]: I1011 10:53:05.656049 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kolla-config\" (UniqueName: \"kubernetes.io/configmap/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-kolla-config\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.656898 master-1 kubenswrapper[4771]: I1011 10:53:05.656844 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"operator-scripts\" (UniqueName: \"kubernetes.io/configmap/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-operator-scripts\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.657642 master-1 kubenswrapper[4771]: I1011 10:53:05.657604 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:53:05.657708 master-1 kubenswrapper[4771]: I1011 10:53:05.657654 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-6ff4ce29-e30c-42fb-a4c8-2f59d5ba9d6a\" (UniqueName: \"kubernetes.io/csi/topolvm.io^050fd1fe-ba08-48fe-9f1b-0beca688083e\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/dba4f94046e1f12f87d8083348b61fdbaae5a90360f23733cc71d53b91882c14/globalmount\"" pod="openstack/openstack-galera-0" Oct 11 10:53:05.658709 master-1 kubenswrapper[4771]: I1011 10:53:05.658659 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-combined-ca-bundle\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.670947 master-1 kubenswrapper[4771]: I1011 10:53:05.670890 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secrets\" (UniqueName: \"kubernetes.io/secret/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-secrets\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.673307 master-1 kubenswrapper[4771]: I1011 10:53:05.673150 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"galera-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-galera-tls-certs\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:05.741275 master-1 kubenswrapper[4771]: I1011 10:53:05.741214 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnm9t\" (UniqueName: \"kubernetes.io/projected/3c0fb436-6e71-4a5a-844e-e8c8e83eacdd-kube-api-access-fnm9t\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:06.730051 master-1 kubenswrapper[4771]: I1011 10:53:06.729995 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-metrics-xvjvs"] Oct 11 10:53:06.759883 master-1 kubenswrapper[4771]: W1011 10:53:06.759746 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0617ee1_7793_4630_9daa_5d4d02f1c5fe.slice/crio-05cb2f5713429f2a19083b5387159f271af13f2214f8228eb364a1855c606d0e WatchSource:0}: Error finding container 05cb2f5713429f2a19083b5387159f271af13f2214f8228eb364a1855c606d0e: Status 404 returned error can't find the container with id 05cb2f5713429f2a19083b5387159f271af13f2214f8228eb364a1855c606d0e Oct 11 10:53:06.873624 master-1 kubenswrapper[4771]: I1011 10:53:06.873557 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-6ff4ce29-e30c-42fb-a4c8-2f59d5ba9d6a\" (UniqueName: \"kubernetes.io/csi/topolvm.io^050fd1fe-ba08-48fe-9f1b-0beca688083e\") pod \"openstack-galera-0\" (UID: \"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd\") " pod="openstack/openstack-galera-0" Oct 11 10:53:06.940757 master-1 kubenswrapper[4771]: I1011 10:53:06.940686 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-ovs-mvzxp" event={"ID":"71b1c323-2ebf-4a37-9327-840d3f04eda1","Type":"ContainerStarted","Data":"b6fb75d7f46e4440adb60340f2b232acddf760ea45aaaef384a768bccc89f0ea"} Oct 11 10:53:06.942061 master-1 kubenswrapper[4771]: I1011 10:53:06.941998 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-metrics-xvjvs" event={"ID":"a0617ee1-7793-4630-9daa-5d4d02f1c5fe","Type":"ContainerStarted","Data":"05cb2f5713429f2a19083b5387159f271af13f2214f8228eb364a1855c606d0e"} Oct 11 10:53:06.944034 master-1 kubenswrapper[4771]: I1011 10:53:06.943985 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" event={"ID":"37015f12-0983-4016-9f76-6d0e3f641f28","Type":"ContainerStarted","Data":"b3e3b46fc901080e41385771d41f43fbd31766c94a2a5ea0b8a9cb3a8c03ad18"} Oct 11 10:53:06.944285 master-1 kubenswrapper[4771]: I1011 10:53:06.944245 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:53:06.987970 master-1 kubenswrapper[4771]: I1011 10:53:06.987907 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/rabbitmq-cell1-server-0"] Oct 11 10:53:07.192282 master-1 kubenswrapper[4771]: I1011 10:53:07.192201 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/openstack-galera-0" Oct 11 10:53:07.244599 master-1 kubenswrapper[4771]: W1011 10:53:07.244534 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831321b9_20ce_409b_8bdb_ec231aef5f35.slice/crio-a626dcc125802c613cc9d03770554cd54066251711845b038bb38fd2f77a5968 WatchSource:0}: Error finding container a626dcc125802c613cc9d03770554cd54066251711845b038bb38fd2f77a5968: Status 404 returned error can't find the container with id a626dcc125802c613cc9d03770554cd54066251711845b038bb38fd2f77a5968 Oct 11 10:53:07.873052 master-1 kubenswrapper[4771]: I1011 10:53:07.872938 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" podStartSLOduration=17.872900991 podStartE2EDuration="17.872900991s" podCreationTimestamp="2025-10-11 10:52:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:53:07.609632453 +0000 UTC m=+1619.583858894" watchObservedRunningTime="2025-10-11 10:53:07.872900991 +0000 UTC m=+1619.847127432" Oct 11 10:53:07.873590 master-1 kubenswrapper[4771]: I1011 10:53:07.873214 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/openstack-galera-0"] Oct 11 10:53:07.907680 master-1 kubenswrapper[4771]: W1011 10:53:07.907621 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c0fb436_6e71_4a5a_844e_e8c8e83eacdd.slice/crio-ca1dcb6fb520857d20bf0b01a4ba1243fb560719ce5a2f79cf56d4478eb48e6f WatchSource:0}: Error finding container ca1dcb6fb520857d20bf0b01a4ba1243fb560719ce5a2f79cf56d4478eb48e6f: Status 404 returned error can't find the container with id ca1dcb6fb520857d20bf0b01a4ba1243fb560719ce5a2f79cf56d4478eb48e6f Oct 11 10:53:07.952336 master-1 kubenswrapper[4771]: I1011 10:53:07.952292 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7" event={"ID":"4ab25521-7fba-40c9-b3db-377b1d0ec7a1","Type":"ContainerStarted","Data":"ab1b69d2441c8c2ff63860b07a0aa63ef1fb0d5772284a7feadcbd7ce53fcb08"} Oct 11 10:53:07.952607 master-1 kubenswrapper[4771]: I1011 10:53:07.952594 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:07.954080 master-1 kubenswrapper[4771]: I1011 10:53:07.954002 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/rabbitmq-server-0" event={"ID":"6fe99fba-e358-4203-a516-04b9ae19d789","Type":"ContainerStarted","Data":"7f8fc71d7ad02d8da77907079a53d04db1a0fb1212260a6e3e48d8f38e321946"} Oct 11 10:53:07.955678 master-1 kubenswrapper[4771]: I1011 10:53:07.955626 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/memcached-0" event={"ID":"3843977f-b9b9-4f98-9205-5dbe3113fa5e","Type":"ContainerStarted","Data":"3e0020e65364a58f1137aff3bd83a82293e06f3c004bc46674cb85c4c72b8928"} Oct 11 10:53:07.955749 master-1 kubenswrapper[4771]: I1011 10:53:07.955722 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/memcached-0" Oct 11 10:53:07.956928 master-1 kubenswrapper[4771]: I1011 10:53:07.956877 4771 generic.go:334] "Generic (PLEG): container finished" podID="71b1c323-2ebf-4a37-9327-840d3f04eda1" containerID="b6fb75d7f46e4440adb60340f2b232acddf760ea45aaaef384a768bccc89f0ea" exitCode=0 Oct 11 10:53:07.956995 master-1 kubenswrapper[4771]: I1011 10:53:07.956923 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-ovs-mvzxp" event={"ID":"71b1c323-2ebf-4a37-9327-840d3f04eda1","Type":"ContainerDied","Data":"b6fb75d7f46e4440adb60340f2b232acddf760ea45aaaef384a768bccc89f0ea"} Oct 11 10:53:07.957896 master-1 kubenswrapper[4771]: I1011 10:53:07.957864 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/openstack-galera-0" event={"ID":"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd","Type":"ContainerStarted","Data":"ca1dcb6fb520857d20bf0b01a4ba1243fb560719ce5a2f79cf56d4478eb48e6f"} Oct 11 10:53:07.959090 master-1 kubenswrapper[4771]: I1011 10:53:07.959065 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/rabbitmq-cell1-server-0" event={"ID":"831321b9-20ce-409b-8bdb-ec231aef5f35","Type":"ContainerStarted","Data":"a626dcc125802c613cc9d03770554cd54066251711845b038bb38fd2f77a5968"} Oct 11 10:53:08.190613 master-1 kubenswrapper[4771]: I1011 10:53:08.190519 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ovn-controller-mtzk7" podStartSLOduration=2.905738598 podStartE2EDuration="7.190496864s" podCreationTimestamp="2025-10-11 10:53:01 +0000 UTC" firstStartedPulling="2025-10-11 10:53:02.071044426 +0000 UTC m=+1614.045270867" lastFinishedPulling="2025-10-11 10:53:06.355802652 +0000 UTC m=+1618.330029133" observedRunningTime="2025-10-11 10:53:08.190210816 +0000 UTC m=+1620.164437267" watchObservedRunningTime="2025-10-11 10:53:08.190496864 +0000 UTC m=+1620.164723305" Oct 11 10:53:08.706094 master-1 kubenswrapper[4771]: I1011 10:53:08.705918 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/memcached-0" podStartSLOduration=8.626226719 podStartE2EDuration="13.705897399s" podCreationTimestamp="2025-10-11 10:52:55 +0000 UTC" firstStartedPulling="2025-10-11 10:53:01.245504935 +0000 UTC m=+1613.219731376" lastFinishedPulling="2025-10-11 10:53:06.325175615 +0000 UTC m=+1618.299402056" observedRunningTime="2025-10-11 10:53:08.702575902 +0000 UTC m=+1620.676802383" watchObservedRunningTime="2025-10-11 10:53:08.705897399 +0000 UTC m=+1620.680123840" Oct 11 10:53:08.983816 master-1 kubenswrapper[4771]: I1011 10:53:08.983685 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-ovs-mvzxp" event={"ID":"71b1c323-2ebf-4a37-9327-840d3f04eda1","Type":"ContainerStarted","Data":"8c9042b1be70f7fdb6a80fca7cde394effe42af7205cc1fcf3a6254bf3330806"} Oct 11 10:53:08.991288 master-1 kubenswrapper[4771]: I1011 10:53:08.991059 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/rabbitmq-cell1-server-0" event={"ID":"831321b9-20ce-409b-8bdb-ec231aef5f35","Type":"ContainerStarted","Data":"46e81e63ab3ceec54c8e0da9448541aeaf71c73eb9783cb511b8ceaa6d4dbd06"} Oct 11 10:53:11.198808 master-1 kubenswrapper[4771]: I1011 10:53:11.198724 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:53:12.210152 master-1 kubenswrapper[4771]: I1011 10:53:12.210073 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-86d565bb9-85bsq"] Oct 11 10:53:12.211308 master-1 kubenswrapper[4771]: I1011 10:53:12.211259 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.273718 master-1 kubenswrapper[4771]: I1011 10:53:12.273626 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-86d565bb9-85bsq"] Oct 11 10:53:12.318524 master-1 kubenswrapper[4771]: I1011 10:53:12.318450 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-dns-svc\") pod \"dnsmasq-dns-86d565bb9-85bsq\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.318788 master-1 kubenswrapper[4771]: I1011 10:53:12.318639 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-config\") pod \"dnsmasq-dns-86d565bb9-85bsq\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.318788 master-1 kubenswrapper[4771]: I1011 10:53:12.318691 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4pv\" (UniqueName: \"kubernetes.io/projected/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-kube-api-access-7b4pv\") pod \"dnsmasq-dns-86d565bb9-85bsq\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.420531 master-1 kubenswrapper[4771]: I1011 10:53:12.420429 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-config\") pod \"dnsmasq-dns-86d565bb9-85bsq\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.420813 master-1 kubenswrapper[4771]: I1011 10:53:12.420790 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4pv\" (UniqueName: \"kubernetes.io/projected/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-kube-api-access-7b4pv\") pod \"dnsmasq-dns-86d565bb9-85bsq\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.421094 master-1 kubenswrapper[4771]: I1011 10:53:12.421075 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-dns-svc\") pod \"dnsmasq-dns-86d565bb9-85bsq\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.421523 master-1 kubenswrapper[4771]: I1011 10:53:12.421473 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-config\") pod \"dnsmasq-dns-86d565bb9-85bsq\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.422096 master-1 kubenswrapper[4771]: I1011 10:53:12.422064 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-dns-svc\") pod \"dnsmasq-dns-86d565bb9-85bsq\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.614189 master-1 kubenswrapper[4771]: I1011 10:53:12.614112 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4pv\" (UniqueName: \"kubernetes.io/projected/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-kube-api-access-7b4pv\") pod \"dnsmasq-dns-86d565bb9-85bsq\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.829034 master-1 kubenswrapper[4771]: I1011 10:53:12.828920 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:12.906576 master-1 kubenswrapper[4771]: I1011 10:53:12.906497 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/openstack-cell1-galera-1"] Oct 11 10:53:12.907863 master-1 kubenswrapper[4771]: I1011 10:53:12.907823 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:12.911841 master-1 kubenswrapper[4771]: I1011 10:53:12.911803 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-galera-openstack-cell1-svc" Oct 11 10:53:12.912128 master-1 kubenswrapper[4771]: I1011 10:53:12.912095 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-cell1-config-data" Oct 11 10:53:12.912825 master-1 kubenswrapper[4771]: I1011 10:53:12.912785 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-cell1-scripts" Oct 11 10:53:13.009390 master-1 kubenswrapper[4771]: I1011 10:53:13.009200 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/openstack-cell1-galera-1"] Oct 11 10:53:13.038896 master-1 kubenswrapper[4771]: I1011 10:53:13.038648 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv8z4\" (UniqueName: \"kubernetes.io/projected/66c0cd85-28ea-42de-8432-8803026d3124-kube-api-access-zv8z4\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.038896 master-1 kubenswrapper[4771]: I1011 10:53:13.038756 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"galera-tls-certs\" (UniqueName: \"kubernetes.io/secret/66c0cd85-28ea-42de-8432-8803026d3124-galera-tls-certs\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.038896 master-1 kubenswrapper[4771]: I1011 10:53:13.038804 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a83a0ef9-545f-41b1-a315-e924a92d6f81\" (UniqueName: \"kubernetes.io/csi/topolvm.io^c48462de-767c-4d18-883d-e2a0b148d485\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.038896 master-1 kubenswrapper[4771]: I1011 10:53:13.038852 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-default\" (UniqueName: \"kubernetes.io/configmap/66c0cd85-28ea-42de-8432-8803026d3124-config-data-default\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.039219 master-1 kubenswrapper[4771]: I1011 10:53:13.038912 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kolla-config\" (UniqueName: \"kubernetes.io/configmap/66c0cd85-28ea-42de-8432-8803026d3124-kolla-config\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.039219 master-1 kubenswrapper[4771]: I1011 10:53:13.038968 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secrets\" (UniqueName: \"kubernetes.io/secret/66c0cd85-28ea-42de-8432-8803026d3124-secrets\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.039219 master-1 kubenswrapper[4771]: I1011 10:53:13.038998 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-scripts\" (UniqueName: \"kubernetes.io/configmap/66c0cd85-28ea-42de-8432-8803026d3124-operator-scripts\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.039219 master-1 kubenswrapper[4771]: I1011 10:53:13.039036 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/66c0cd85-28ea-42de-8432-8803026d3124-combined-ca-bundle\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.039219 master-1 kubenswrapper[4771]: I1011 10:53:13.039072 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-generated\" (UniqueName: \"kubernetes.io/empty-dir/66c0cd85-28ea-42de-8432-8803026d3124-config-data-generated\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.143427 master-1 kubenswrapper[4771]: I1011 10:53:13.143383 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kolla-config\" (UniqueName: \"kubernetes.io/configmap/66c0cd85-28ea-42de-8432-8803026d3124-kolla-config\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.143640 master-1 kubenswrapper[4771]: I1011 10:53:13.143442 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secrets\" (UniqueName: \"kubernetes.io/secret/66c0cd85-28ea-42de-8432-8803026d3124-secrets\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.143640 master-1 kubenswrapper[4771]: I1011 10:53:13.143464 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"operator-scripts\" (UniqueName: \"kubernetes.io/configmap/66c0cd85-28ea-42de-8432-8803026d3124-operator-scripts\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.143640 master-1 kubenswrapper[4771]: I1011 10:53:13.143487 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/66c0cd85-28ea-42de-8432-8803026d3124-combined-ca-bundle\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.143640 master-1 kubenswrapper[4771]: I1011 10:53:13.143509 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-generated\" (UniqueName: \"kubernetes.io/empty-dir/66c0cd85-28ea-42de-8432-8803026d3124-config-data-generated\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.143640 master-1 kubenswrapper[4771]: I1011 10:53:13.143568 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-zv8z4\" (UniqueName: \"kubernetes.io/projected/66c0cd85-28ea-42de-8432-8803026d3124-kube-api-access-zv8z4\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.143640 master-1 kubenswrapper[4771]: I1011 10:53:13.143597 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"galera-tls-certs\" (UniqueName: \"kubernetes.io/secret/66c0cd85-28ea-42de-8432-8803026d3124-galera-tls-certs\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.143640 master-1 kubenswrapper[4771]: I1011 10:53:13.143625 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-a83a0ef9-545f-41b1-a315-e924a92d6f81\" (UniqueName: \"kubernetes.io/csi/topolvm.io^c48462de-767c-4d18-883d-e2a0b148d485\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.143640 master-1 kubenswrapper[4771]: I1011 10:53:13.143647 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-default\" (UniqueName: \"kubernetes.io/configmap/66c0cd85-28ea-42de-8432-8803026d3124-config-data-default\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.144233 master-1 kubenswrapper[4771]: I1011 10:53:13.144131 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-generated\" (UniqueName: \"kubernetes.io/empty-dir/66c0cd85-28ea-42de-8432-8803026d3124-config-data-generated\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.146253 master-1 kubenswrapper[4771]: I1011 10:53:13.146206 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kolla-config\" (UniqueName: \"kubernetes.io/configmap/66c0cd85-28ea-42de-8432-8803026d3124-kolla-config\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.146911 master-1 kubenswrapper[4771]: I1011 10:53:13.146874 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:53:13.146975 master-1 kubenswrapper[4771]: I1011 10:53:13.146921 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-a83a0ef9-545f-41b1-a315-e924a92d6f81\" (UniqueName: \"kubernetes.io/csi/topolvm.io^c48462de-767c-4d18-883d-e2a0b148d485\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/937b2836a25f50d4b06480c0eec40bfcaffaa9f58948ca2aaf81b69d8b3600d7/globalmount\"" pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.148641 master-1 kubenswrapper[4771]: I1011 10:53:13.148601 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-default\" (UniqueName: \"kubernetes.io/configmap/66c0cd85-28ea-42de-8432-8803026d3124-config-data-default\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.150461 master-1 kubenswrapper[4771]: I1011 10:53:13.150432 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"operator-scripts\" (UniqueName: \"kubernetes.io/configmap/66c0cd85-28ea-42de-8432-8803026d3124-operator-scripts\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.153668 master-1 kubenswrapper[4771]: I1011 10:53:13.153645 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"galera-tls-certs\" (UniqueName: \"kubernetes.io/secret/66c0cd85-28ea-42de-8432-8803026d3124-galera-tls-certs\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.153800 master-1 kubenswrapper[4771]: I1011 10:53:13.153740 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secrets\" (UniqueName: \"kubernetes.io/secret/66c0cd85-28ea-42de-8432-8803026d3124-secrets\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.154343 master-1 kubenswrapper[4771]: I1011 10:53:13.154304 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/66c0cd85-28ea-42de-8432-8803026d3124-combined-ca-bundle\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.171436 master-1 kubenswrapper[4771]: I1011 10:53:13.169780 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv8z4\" (UniqueName: \"kubernetes.io/projected/66c0cd85-28ea-42de-8432-8803026d3124-kube-api-access-zv8z4\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:13.411103 master-1 kubenswrapper[4771]: I1011 10:53:13.411022 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-86d565bb9-85bsq"] Oct 11 10:53:13.425883 master-1 kubenswrapper[4771]: W1011 10:53:13.425813 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ae0f8e3_9e87_45b8_8313_a0b65cf33106.slice/crio-1af26e0c3df30b750253930a520b32ea24880275e960e13e9910257c86f202ff WatchSource:0}: Error finding container 1af26e0c3df30b750253930a520b32ea24880275e960e13e9910257c86f202ff: Status 404 returned error can't find the container with id 1af26e0c3df30b750253930a520b32ea24880275e960e13e9910257c86f202ff Oct 11 10:53:14.047670 master-1 kubenswrapper[4771]: I1011 10:53:14.047569 4771 generic.go:334] "Generic (PLEG): container finished" podID="0ae0f8e3-9e87-45b8-8313-a0b65cf33106" containerID="fd43d772f12b2955515b0207673e261220c08cfd99b72815c0e4dd5a30cfab8c" exitCode=0 Oct 11 10:53:14.048327 master-1 kubenswrapper[4771]: I1011 10:53:14.047764 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" event={"ID":"0ae0f8e3-9e87-45b8-8313-a0b65cf33106","Type":"ContainerDied","Data":"fd43d772f12b2955515b0207673e261220c08cfd99b72815c0e4dd5a30cfab8c"} Oct 11 10:53:14.048546 master-1 kubenswrapper[4771]: I1011 10:53:14.048339 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" event={"ID":"0ae0f8e3-9e87-45b8-8313-a0b65cf33106","Type":"ContainerStarted","Data":"1af26e0c3df30b750253930a520b32ea24880275e960e13e9910257c86f202ff"} Oct 11 10:53:14.053460 master-1 kubenswrapper[4771]: I1011 10:53:14.053351 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-ovs-mvzxp" event={"ID":"71b1c323-2ebf-4a37-9327-840d3f04eda1","Type":"ContainerStarted","Data":"63099c14644e997d6b7af177d0bc5a9cb1666fc0b8bfdde4fc39c2515570d309"} Oct 11 10:53:14.053752 master-1 kubenswrapper[4771]: I1011 10:53:14.053668 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:14.056075 master-1 kubenswrapper[4771]: I1011 10:53:14.055274 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/openstack-galera-0" event={"ID":"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd","Type":"ContainerStarted","Data":"d123df26764927b959f6821afc87dfe457d42e15c12942ada27bc877e1b79e1d"} Oct 11 10:53:14.061723 master-1 kubenswrapper[4771]: I1011 10:53:14.058298 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-metrics-xvjvs" event={"ID":"a0617ee1-7793-4630-9daa-5d4d02f1c5fe","Type":"ContainerStarted","Data":"880b4d3b6aa1c158a513f02fc144ecdba62fe2fbc3a04f13a4b6bff459d1e020"} Oct 11 10:53:14.155091 master-1 kubenswrapper[4771]: I1011 10:53:14.154975 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ovn-controller-metrics-xvjvs" podStartSLOduration=5.980741034 podStartE2EDuration="12.154948073s" podCreationTimestamp="2025-10-11 10:53:02 +0000 UTC" firstStartedPulling="2025-10-11 10:53:06.762219839 +0000 UTC m=+1618.736446280" lastFinishedPulling="2025-10-11 10:53:12.936426848 +0000 UTC m=+1624.910653319" observedRunningTime="2025-10-11 10:53:14.150926496 +0000 UTC m=+1626.125152997" watchObservedRunningTime="2025-10-11 10:53:14.154948073 +0000 UTC m=+1626.129174554" Oct 11 10:53:14.199056 master-1 kubenswrapper[4771]: I1011 10:53:14.188545 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ovn-controller-ovs-mvzxp" podStartSLOduration=9.46569798 podStartE2EDuration="13.188525973s" podCreationTimestamp="2025-10-11 10:53:01 +0000 UTC" firstStartedPulling="2025-10-11 10:53:02.617670826 +0000 UTC m=+1614.591897267" lastFinishedPulling="2025-10-11 10:53:06.340498769 +0000 UTC m=+1618.314725260" observedRunningTime="2025-10-11 10:53:14.186565447 +0000 UTC m=+1626.160791928" watchObservedRunningTime="2025-10-11 10:53:14.188525973 +0000 UTC m=+1626.162752414" Oct 11 10:53:15.077173 master-1 kubenswrapper[4771]: I1011 10:53:15.077119 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" event={"ID":"0ae0f8e3-9e87-45b8-8313-a0b65cf33106","Type":"ContainerStarted","Data":"14ed7d218f9217fbceb4436ac3f26fb55858bf77044f44bd18a2d4ffe4eacee3"} Oct 11 10:53:15.077664 master-1 kubenswrapper[4771]: I1011 10:53:15.077586 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:15.078189 master-1 kubenswrapper[4771]: I1011 10:53:15.078131 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:15.088924 master-1 kubenswrapper[4771]: I1011 10:53:15.088863 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-a83a0ef9-545f-41b1-a315-e924a92d6f81\" (UniqueName: \"kubernetes.io/csi/topolvm.io^c48462de-767c-4d18-883d-e2a0b148d485\") pod \"openstack-cell1-galera-1\" (UID: \"66c0cd85-28ea-42de-8432-8803026d3124\") " pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:15.108726 master-1 kubenswrapper[4771]: I1011 10:53:15.108637 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" podStartSLOduration=3.108616252 podStartE2EDuration="3.108616252s" podCreationTimestamp="2025-10-11 10:53:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:53:15.102887367 +0000 UTC m=+1627.077113838" watchObservedRunningTime="2025-10-11 10:53:15.108616252 +0000 UTC m=+1627.082842703" Oct 11 10:53:15.336965 master-1 kubenswrapper[4771]: I1011 10:53:15.336855 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:15.471543 master-1 kubenswrapper[4771]: I1011 10:53:15.471436 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/prometheus-metric-storage-0"] Oct 11 10:53:15.475906 master-1 kubenswrapper[4771]: I1011 10:53:15.475812 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.482272 master-1 kubenswrapper[4771]: I1011 10:53:15.479262 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"prometheus-metric-storage-thanos-prometheus-http-client-file" Oct 11 10:53:15.482272 master-1 kubenswrapper[4771]: I1011 10:53:15.479722 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"prometheus-metric-storage-web-config" Oct 11 10:53:15.482272 master-1 kubenswrapper[4771]: I1011 10:53:15.479720 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"prometheus-metric-storage-rulefiles-0" Oct 11 10:53:15.482272 master-1 kubenswrapper[4771]: I1011 10:53:15.480945 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"prometheus-metric-storage" Oct 11 10:53:15.489917 master-1 kubenswrapper[4771]: I1011 10:53:15.489428 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"prometheus-metric-storage-tls-assets-0" Oct 11 10:53:15.494266 master-1 kubenswrapper[4771]: I1011 10:53:15.493342 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/prometheus-metric-storage-0"] Oct 11 10:53:15.527176 master-1 kubenswrapper[4771]: I1011 10:53:15.527111 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/memcached-0" Oct 11 10:53:15.614130 master-1 kubenswrapper[4771]: I1011 10:53:15.613928 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-tls-assets\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.614130 master-1 kubenswrapper[4771]: I1011 10:53:15.614002 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-metric-storage-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-prometheus-metric-storage-rulefiles-0\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.614469 master-1 kubenswrapper[4771]: I1011 10:53:15.614172 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-web-config\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.614469 master-1 kubenswrapper[4771]: I1011 10:53:15.614342 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-thanos-prometheus-http-client-file\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.614469 master-1 kubenswrapper[4771]: I1011 10:53:15.614459 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config-out\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.614618 master-1 kubenswrapper[4771]: I1011 10:53:15.614495 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.614618 master-1 kubenswrapper[4771]: I1011 10:53:15.614522 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq956\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-kube-api-access-lq956\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.615242 master-1 kubenswrapper[4771]: I1011 10:53:15.614984 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.719730 master-1 kubenswrapper[4771]: I1011 10:53:15.716964 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-web-config\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.719730 master-1 kubenswrapper[4771]: I1011 10:53:15.717030 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-thanos-prometheus-http-client-file\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.719730 master-1 kubenswrapper[4771]: I1011 10:53:15.717066 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config-out\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.719730 master-1 kubenswrapper[4771]: I1011 10:53:15.717097 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.719730 master-1 kubenswrapper[4771]: I1011 10:53:15.717117 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-lq956\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-kube-api-access-lq956\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.719730 master-1 kubenswrapper[4771]: I1011 10:53:15.717145 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.719730 master-1 kubenswrapper[4771]: I1011 10:53:15.717178 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-tls-assets\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.719730 master-1 kubenswrapper[4771]: I1011 10:53:15.717202 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"prometheus-metric-storage-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-prometheus-metric-storage-rulefiles-0\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.719730 master-1 kubenswrapper[4771]: I1011 10:53:15.718645 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"prometheus-metric-storage-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-prometheus-metric-storage-rulefiles-0\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.720677 master-1 kubenswrapper[4771]: I1011 10:53:15.720626 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:53:15.720747 master-1 kubenswrapper[4771]: I1011 10:53:15.720697 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/37699e2b75858a97c8af891d5e1a76727de9abb22a62dc041bfd38b0b8d8c160/globalmount\"" pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.720929 master-1 kubenswrapper[4771]: I1011 10:53:15.720876 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-thanos-prometheus-http-client-file\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.721889 master-1 kubenswrapper[4771]: I1011 10:53:15.721844 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.723094 master-1 kubenswrapper[4771]: I1011 10:53:15.722850 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-tls-assets\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.723201 master-1 kubenswrapper[4771]: I1011 10:53:15.723167 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-web-config\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.728831 master-1 kubenswrapper[4771]: I1011 10:53:15.728775 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config-out\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.743822 master-1 kubenswrapper[4771]: I1011 10:53:15.743720 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq956\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-kube-api-access-lq956\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:15.847790 master-1 kubenswrapper[4771]: I1011 10:53:15.847722 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/openstack-cell1-galera-1"] Oct 11 10:53:16.090563 master-1 kubenswrapper[4771]: I1011 10:53:16.090452 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/openstack-cell1-galera-1" event={"ID":"66c0cd85-28ea-42de-8432-8803026d3124","Type":"ContainerStarted","Data":"a314c1f4c176eb2e0cd74daae6e743d6300a11be6408bfc38cc9f82fcaeaf8f6"} Oct 11 10:53:16.090563 master-1 kubenswrapper[4771]: I1011 10:53:16.090536 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/openstack-cell1-galera-1" event={"ID":"66c0cd85-28ea-42de-8432-8803026d3124","Type":"ContainerStarted","Data":"987c66295a0779686f133c30ffd38495bdaeed6a9bd1459e540ed23e5b590709"} Oct 11 10:53:17.725101 master-1 kubenswrapper[4771]: I1011 10:53:17.725013 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") pod \"prometheus-metric-storage-0\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:17.832568 master-1 kubenswrapper[4771]: I1011 10:53:17.832515 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovsdbserver-nb-1"] Oct 11 10:53:17.835041 master-1 kubenswrapper[4771]: I1011 10:53:17.835017 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:17.838541 master-1 kubenswrapper[4771]: I1011 10:53:17.838472 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ovndbcluster-nb-ovndbs" Oct 11 10:53:17.839024 master-1 kubenswrapper[4771]: I1011 10:53:17.838954 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovndbcluster-nb-scripts" Oct 11 10:53:17.847126 master-1 kubenswrapper[4771]: I1011 10:53:17.846975 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovsdbserver-nb-1"] Oct 11 10:53:17.854153 master-1 kubenswrapper[4771]: I1011 10:53:17.853825 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovndbcluster-nb-config" Oct 11 10:53:17.898198 master-1 kubenswrapper[4771]: I1011 10:53:17.898127 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:17.972915 master-1 kubenswrapper[4771]: I1011 10:53:17.972800 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8f9b018c-eb14-4c27-adcb-ba613238c78b-combined-ca-bundle\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:17.972915 master-1 kubenswrapper[4771]: I1011 10:53:17.972883 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f9b018c-eb14-4c27-adcb-ba613238c78b-config\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:17.973401 master-1 kubenswrapper[4771]: I1011 10:53:17.973223 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/8f9b018c-eb14-4c27-adcb-ba613238c78b-scripts\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:17.973401 master-1 kubenswrapper[4771]: I1011 10:53:17.973309 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rkx\" (UniqueName: \"kubernetes.io/projected/8f9b018c-eb14-4c27-adcb-ba613238c78b-kube-api-access-t6rkx\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:17.973401 master-1 kubenswrapper[4771]: I1011 10:53:17.973382 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9b018c-eb14-4c27-adcb-ba613238c78b-ovsdbserver-nb-tls-certs\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:17.973558 master-1 kubenswrapper[4771]: I1011 10:53:17.973420 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdb-rundir\" (UniqueName: \"kubernetes.io/empty-dir/8f9b018c-eb14-4c27-adcb-ba613238c78b-ovsdb-rundir\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:17.973558 master-1 kubenswrapper[4771]: I1011 10:53:17.973494 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9b018c-eb14-4c27-adcb-ba613238c78b-metrics-certs-tls-certs\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:17.973558 master-1 kubenswrapper[4771]: I1011 10:53:17.973545 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ca866366-3239-46ad-903c-e44f9f2ec0e3\" (UniqueName: \"kubernetes.io/csi/topolvm.io^f06a64ac-0910-4092-b155-298c9316de58\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.075640 master-1 kubenswrapper[4771]: I1011 10:53:18.075529 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-ca866366-3239-46ad-903c-e44f9f2ec0e3\" (UniqueName: \"kubernetes.io/csi/topolvm.io^f06a64ac-0910-4092-b155-298c9316de58\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.075640 master-1 kubenswrapper[4771]: I1011 10:53:18.075635 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8f9b018c-eb14-4c27-adcb-ba613238c78b-combined-ca-bundle\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.075640 master-1 kubenswrapper[4771]: I1011 10:53:18.075662 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f9b018c-eb14-4c27-adcb-ba613238c78b-config\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.076052 master-1 kubenswrapper[4771]: I1011 10:53:18.075741 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/8f9b018c-eb14-4c27-adcb-ba613238c78b-scripts\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.076052 master-1 kubenswrapper[4771]: I1011 10:53:18.075770 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rkx\" (UniqueName: \"kubernetes.io/projected/8f9b018c-eb14-4c27-adcb-ba613238c78b-kube-api-access-t6rkx\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.076052 master-1 kubenswrapper[4771]: I1011 10:53:18.075798 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9b018c-eb14-4c27-adcb-ba613238c78b-ovsdbserver-nb-tls-certs\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.076052 master-1 kubenswrapper[4771]: I1011 10:53:18.075826 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdb-rundir\" (UniqueName: \"kubernetes.io/empty-dir/8f9b018c-eb14-4c27-adcb-ba613238c78b-ovsdb-rundir\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.076052 master-1 kubenswrapper[4771]: I1011 10:53:18.075855 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9b018c-eb14-4c27-adcb-ba613238c78b-metrics-certs-tls-certs\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.077806 master-1 kubenswrapper[4771]: I1011 10:53:18.077734 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdb-rundir\" (UniqueName: \"kubernetes.io/empty-dir/8f9b018c-eb14-4c27-adcb-ba613238c78b-ovsdb-rundir\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.078615 master-1 kubenswrapper[4771]: I1011 10:53:18.078540 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:53:18.078783 master-1 kubenswrapper[4771]: I1011 10:53:18.078626 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-ca866366-3239-46ad-903c-e44f9f2ec0e3\" (UniqueName: \"kubernetes.io/csi/topolvm.io^f06a64ac-0910-4092-b155-298c9316de58\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/3fdbc2eaaf604b357bf2e0714ac3de7e5d4c02faf07fa6a0914801badbeb5a83/globalmount\"" pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.080287 master-1 kubenswrapper[4771]: I1011 10:53:18.080204 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9b018c-eb14-4c27-adcb-ba613238c78b-metrics-certs-tls-certs\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.081218 master-1 kubenswrapper[4771]: I1011 10:53:18.081130 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f9b018c-eb14-4c27-adcb-ba613238c78b-config\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.081713 master-1 kubenswrapper[4771]: I1011 10:53:18.081654 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/8f9b018c-eb14-4c27-adcb-ba613238c78b-scripts\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.084308 master-1 kubenswrapper[4771]: I1011 10:53:18.084242 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8f9b018c-eb14-4c27-adcb-ba613238c78b-combined-ca-bundle\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.084308 master-1 kubenswrapper[4771]: I1011 10:53:18.084266 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9b018c-eb14-4c27-adcb-ba613238c78b-ovsdbserver-nb-tls-certs\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.109735 master-1 kubenswrapper[4771]: I1011 10:53:18.109655 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rkx\" (UniqueName: \"kubernetes.io/projected/8f9b018c-eb14-4c27-adcb-ba613238c78b-kube-api-access-t6rkx\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:18.128441 master-1 kubenswrapper[4771]: I1011 10:53:18.122023 4771 generic.go:334] "Generic (PLEG): container finished" podID="3c0fb436-6e71-4a5a-844e-e8c8e83eacdd" containerID="d123df26764927b959f6821afc87dfe457d42e15c12942ada27bc877e1b79e1d" exitCode=0 Oct 11 10:53:18.128441 master-1 kubenswrapper[4771]: I1011 10:53:18.122093 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/openstack-galera-0" event={"ID":"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd","Type":"ContainerDied","Data":"d123df26764927b959f6821afc87dfe457d42e15c12942ada27bc877e1b79e1d"} Oct 11 10:53:18.445173 master-1 kubenswrapper[4771]: W1011 10:53:18.445113 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c30d1f_16d5_4106_bfae_e6c2d2f64f13.slice/crio-b089e8e4b13a2f28ac29e4b38aaf2c96910827eb74888a884dedec830020d9fe WatchSource:0}: Error finding container b089e8e4b13a2f28ac29e4b38aaf2c96910827eb74888a884dedec830020d9fe: Status 404 returned error can't find the container with id b089e8e4b13a2f28ac29e4b38aaf2c96910827eb74888a884dedec830020d9fe Oct 11 10:53:18.445983 master-1 kubenswrapper[4771]: I1011 10:53:18.445881 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/prometheus-metric-storage-0"] Oct 11 10:53:19.131960 master-1 kubenswrapper[4771]: I1011 10:53:19.131783 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/openstack-galera-0" event={"ID":"3c0fb436-6e71-4a5a-844e-e8c8e83eacdd","Type":"ContainerStarted","Data":"75ee61d883ca29a4a72af36026e1ef45716f46c2f0b47656f0af04d68b4ea69b"} Oct 11 10:53:19.134920 master-1 kubenswrapper[4771]: I1011 10:53:19.134878 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerStarted","Data":"b089e8e4b13a2f28ac29e4b38aaf2c96910827eb74888a884dedec830020d9fe"} Oct 11 10:53:19.577613 master-1 kubenswrapper[4771]: I1011 10:53:19.575829 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-ca866366-3239-46ad-903c-e44f9f2ec0e3\" (UniqueName: \"kubernetes.io/csi/topolvm.io^f06a64ac-0910-4092-b155-298c9316de58\") pod \"ovsdbserver-nb-1\" (UID: \"8f9b018c-eb14-4c27-adcb-ba613238c78b\") " pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:19.671377 master-1 kubenswrapper[4771]: I1011 10:53:19.671237 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:20.220193 master-1 kubenswrapper[4771]: I1011 10:53:20.220102 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/openstack-galera-0" podStartSLOduration=23.171684584 podStartE2EDuration="28.220082001s" podCreationTimestamp="2025-10-11 10:52:52 +0000 UTC" firstStartedPulling="2025-10-11 10:53:07.910333706 +0000 UTC m=+1619.884560147" lastFinishedPulling="2025-10-11 10:53:12.958731123 +0000 UTC m=+1624.932957564" observedRunningTime="2025-10-11 10:53:19.163291351 +0000 UTC m=+1631.137517792" watchObservedRunningTime="2025-10-11 10:53:20.220082001 +0000 UTC m=+1632.194308442" Oct 11 10:53:20.222779 master-1 kubenswrapper[4771]: I1011 10:53:20.222746 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovsdbserver-nb-1"] Oct 11 10:53:20.238404 master-1 kubenswrapper[4771]: W1011 10:53:20.229050 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f9b018c_eb14_4c27_adcb_ba613238c78b.slice/crio-3d39ea0144914b5f9427ce6ad71ccee33c30187fbaa0472d6fa81f99b090257f WatchSource:0}: Error finding container 3d39ea0144914b5f9427ce6ad71ccee33c30187fbaa0472d6fa81f99b090257f: Status 404 returned error can't find the container with id 3d39ea0144914b5f9427ce6ad71ccee33c30187fbaa0472d6fa81f99b090257f Oct 11 10:53:21.081157 master-1 kubenswrapper[4771]: I1011 10:53:21.081088 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovsdbserver-sb-1"] Oct 11 10:53:21.082530 master-1 kubenswrapper[4771]: I1011 10:53:21.082506 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.086554 master-1 kubenswrapper[4771]: I1011 10:53:21.086488 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovndbcluster-sb-config" Oct 11 10:53:21.086800 master-1 kubenswrapper[4771]: I1011 10:53:21.086697 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ovndbcluster-sb-ovndbs" Oct 11 10:53:21.086941 master-1 kubenswrapper[4771]: I1011 10:53:21.086916 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovndbcluster-sb-scripts" Oct 11 10:53:21.101783 master-1 kubenswrapper[4771]: I1011 10:53:21.101725 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovsdbserver-sb-1"] Oct 11 10:53:21.141572 master-1 kubenswrapper[4771]: I1011 10:53:21.141489 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/c6e032cb-3993-4b10-afe4-7f77821c8583-scripts\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.141572 master-1 kubenswrapper[4771]: I1011 10:53:21.141558 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e032cb-3993-4b10-afe4-7f77821c8583-ovsdbserver-sb-tls-certs\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.141798 master-1 kubenswrapper[4771]: I1011 10:53:21.141601 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdb-rundir\" (UniqueName: \"kubernetes.io/empty-dir/c6e032cb-3993-4b10-afe4-7f77821c8583-ovsdb-rundir\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.141798 master-1 kubenswrapper[4771]: I1011 10:53:21.141634 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e032cb-3993-4b10-afe4-7f77821c8583-metrics-certs-tls-certs\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.141798 master-1 kubenswrapper[4771]: I1011 10:53:21.141670 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2cc50c9a-367c-43a5-9e92-aa42989ee215\" (UniqueName: \"kubernetes.io/csi/topolvm.io^a44ae47b-cc90-4016-a02d-1f3945269ee7\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.141798 master-1 kubenswrapper[4771]: I1011 10:53:21.141692 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6e032cb-3993-4b10-afe4-7f77821c8583-config\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.141798 master-1 kubenswrapper[4771]: I1011 10:53:21.141721 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24vl\" (UniqueName: \"kubernetes.io/projected/c6e032cb-3993-4b10-afe4-7f77821c8583-kube-api-access-v24vl\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.141798 master-1 kubenswrapper[4771]: I1011 10:53:21.141749 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6e032cb-3993-4b10-afe4-7f77821c8583-combined-ca-bundle\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.149557 master-1 kubenswrapper[4771]: I1011 10:53:21.149470 4771 generic.go:334] "Generic (PLEG): container finished" podID="66c0cd85-28ea-42de-8432-8803026d3124" containerID="a314c1f4c176eb2e0cd74daae6e743d6300a11be6408bfc38cc9f82fcaeaf8f6" exitCode=0 Oct 11 10:53:21.149615 master-1 kubenswrapper[4771]: I1011 10:53:21.149561 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/openstack-cell1-galera-1" event={"ID":"66c0cd85-28ea-42de-8432-8803026d3124","Type":"ContainerDied","Data":"a314c1f4c176eb2e0cd74daae6e743d6300a11be6408bfc38cc9f82fcaeaf8f6"} Oct 11 10:53:21.151026 master-1 kubenswrapper[4771]: I1011 10:53:21.150621 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovsdbserver-nb-1" event={"ID":"8f9b018c-eb14-4c27-adcb-ba613238c78b","Type":"ContainerStarted","Data":"3d39ea0144914b5f9427ce6ad71ccee33c30187fbaa0472d6fa81f99b090257f"} Oct 11 10:53:21.243594 master-1 kubenswrapper[4771]: I1011 10:53:21.243527 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e032cb-3993-4b10-afe4-7f77821c8583-metrics-certs-tls-certs\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.244160 master-1 kubenswrapper[4771]: I1011 10:53:21.243616 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-2cc50c9a-367c-43a5-9e92-aa42989ee215\" (UniqueName: \"kubernetes.io/csi/topolvm.io^a44ae47b-cc90-4016-a02d-1f3945269ee7\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.244160 master-1 kubenswrapper[4771]: I1011 10:53:21.243651 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6e032cb-3993-4b10-afe4-7f77821c8583-config\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.244160 master-1 kubenswrapper[4771]: I1011 10:53:21.243687 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-v24vl\" (UniqueName: \"kubernetes.io/projected/c6e032cb-3993-4b10-afe4-7f77821c8583-kube-api-access-v24vl\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.244160 master-1 kubenswrapper[4771]: I1011 10:53:21.243724 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6e032cb-3993-4b10-afe4-7f77821c8583-combined-ca-bundle\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.244160 master-1 kubenswrapper[4771]: I1011 10:53:21.243759 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/c6e032cb-3993-4b10-afe4-7f77821c8583-scripts\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.244160 master-1 kubenswrapper[4771]: I1011 10:53:21.243780 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e032cb-3993-4b10-afe4-7f77821c8583-ovsdbserver-sb-tls-certs\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.244160 master-1 kubenswrapper[4771]: I1011 10:53:21.243818 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdb-rundir\" (UniqueName: \"kubernetes.io/empty-dir/c6e032cb-3993-4b10-afe4-7f77821c8583-ovsdb-rundir\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.244482 master-1 kubenswrapper[4771]: I1011 10:53:21.244379 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdb-rundir\" (UniqueName: \"kubernetes.io/empty-dir/c6e032cb-3993-4b10-afe4-7f77821c8583-ovsdb-rundir\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.245836 master-1 kubenswrapper[4771]: I1011 10:53:21.245779 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6e032cb-3993-4b10-afe4-7f77821c8583-config\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.246272 master-1 kubenswrapper[4771]: I1011 10:53:21.246231 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/c6e032cb-3993-4b10-afe4-7f77821c8583-scripts\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.247000 master-1 kubenswrapper[4771]: I1011 10:53:21.246956 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:53:21.247220 master-1 kubenswrapper[4771]: I1011 10:53:21.247074 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-2cc50c9a-367c-43a5-9e92-aa42989ee215\" (UniqueName: \"kubernetes.io/csi/topolvm.io^a44ae47b-cc90-4016-a02d-1f3945269ee7\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/18d97a7788bd76d183e89e62f30289e4ce86d7e202c647c24a185e51515d686f/globalmount\"" pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.256449 master-1 kubenswrapper[4771]: I1011 10:53:21.249642 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6e032cb-3993-4b10-afe4-7f77821c8583-combined-ca-bundle\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.256449 master-1 kubenswrapper[4771]: I1011 10:53:21.251238 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e032cb-3993-4b10-afe4-7f77821c8583-ovsdbserver-sb-tls-certs\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.256449 master-1 kubenswrapper[4771]: I1011 10:53:21.251332 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e032cb-3993-4b10-afe4-7f77821c8583-metrics-certs-tls-certs\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:21.277330 master-1 kubenswrapper[4771]: I1011 10:53:21.277299 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24vl\" (UniqueName: \"kubernetes.io/projected/c6e032cb-3993-4b10-afe4-7f77821c8583-kube-api-access-v24vl\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:22.701013 master-1 kubenswrapper[4771]: I1011 10:53:22.700949 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-2cc50c9a-367c-43a5-9e92-aa42989ee215\" (UniqueName: \"kubernetes.io/csi/topolvm.io^a44ae47b-cc90-4016-a02d-1f3945269ee7\") pod \"ovsdbserver-sb-1\" (UID: \"c6e032cb-3993-4b10-afe4-7f77821c8583\") " pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:22.833608 master-1 kubenswrapper[4771]: I1011 10:53:22.830668 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:22.914804 master-1 kubenswrapper[4771]: I1011 10:53:22.914746 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:22.919215 master-1 kubenswrapper[4771]: I1011 10:53:22.919133 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-5b4bcc4d85-hc9b5"] Oct 11 10:53:22.919543 master-1 kubenswrapper[4771]: I1011 10:53:22.919495 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" podUID="37015f12-0983-4016-9f76-6d0e3f641f28" containerName="dnsmasq-dns" containerID="cri-o://b3e3b46fc901080e41385771d41f43fbd31766c94a2a5ea0b8a9cb3a8c03ad18" gracePeriod=10 Oct 11 10:53:23.169871 master-1 kubenswrapper[4771]: I1011 10:53:23.169803 4771 generic.go:334] "Generic (PLEG): container finished" podID="37015f12-0983-4016-9f76-6d0e3f641f28" containerID="b3e3b46fc901080e41385771d41f43fbd31766c94a2a5ea0b8a9cb3a8c03ad18" exitCode=0 Oct 11 10:53:23.169871 master-1 kubenswrapper[4771]: I1011 10:53:23.169866 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" event={"ID":"37015f12-0983-4016-9f76-6d0e3f641f28","Type":"ContainerDied","Data":"b3e3b46fc901080e41385771d41f43fbd31766c94a2a5ea0b8a9cb3a8c03ad18"} Oct 11 10:53:24.787379 master-1 kubenswrapper[4771]: I1011 10:53:24.787299 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:53:24.976275 master-1 kubenswrapper[4771]: I1011 10:53:24.976216 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-dns-svc\") pod \"37015f12-0983-4016-9f76-6d0e3f641f28\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " Oct 11 10:53:24.976409 master-1 kubenswrapper[4771]: I1011 10:53:24.976344 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r45c\" (UniqueName: \"kubernetes.io/projected/37015f12-0983-4016-9f76-6d0e3f641f28-kube-api-access-8r45c\") pod \"37015f12-0983-4016-9f76-6d0e3f641f28\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " Oct 11 10:53:24.976639 master-1 kubenswrapper[4771]: I1011 10:53:24.976600 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-config\") pod \"37015f12-0983-4016-9f76-6d0e3f641f28\" (UID: \"37015f12-0983-4016-9f76-6d0e3f641f28\") " Oct 11 10:53:24.980629 master-1 kubenswrapper[4771]: I1011 10:53:24.980527 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37015f12-0983-4016-9f76-6d0e3f641f28-kube-api-access-8r45c" (OuterVolumeSpecName: "kube-api-access-8r45c") pod "37015f12-0983-4016-9f76-6d0e3f641f28" (UID: "37015f12-0983-4016-9f76-6d0e3f641f28"). InnerVolumeSpecName "kube-api-access-8r45c". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:53:25.020713 master-1 kubenswrapper[4771]: I1011 10:53:25.020649 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-config" (OuterVolumeSpecName: "config") pod "37015f12-0983-4016-9f76-6d0e3f641f28" (UID: "37015f12-0983-4016-9f76-6d0e3f641f28"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:53:25.032992 master-1 kubenswrapper[4771]: I1011 10:53:25.032945 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "37015f12-0983-4016-9f76-6d0e3f641f28" (UID: "37015f12-0983-4016-9f76-6d0e3f641f28"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:53:25.083963 master-1 kubenswrapper[4771]: I1011 10:53:25.083900 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-8r45c\" (UniqueName: \"kubernetes.io/projected/37015f12-0983-4016-9f76-6d0e3f641f28-kube-api-access-8r45c\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:25.083963 master-1 kubenswrapper[4771]: I1011 10:53:25.083966 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:25.084147 master-1 kubenswrapper[4771]: I1011 10:53:25.083983 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/37015f12-0983-4016-9f76-6d0e3f641f28-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:25.144658 master-1 kubenswrapper[4771]: I1011 10:53:25.144585 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovsdbserver-sb-1"] Oct 11 10:53:25.153500 master-1 kubenswrapper[4771]: W1011 10:53:25.153435 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6e032cb_3993_4b10_afe4_7f77821c8583.slice/crio-f72206a2629114d93d81b6f349520e61cc1f1bea049e521be9a6c27daec9927c WatchSource:0}: Error finding container f72206a2629114d93d81b6f349520e61cc1f1bea049e521be9a6c27daec9927c: Status 404 returned error can't find the container with id f72206a2629114d93d81b6f349520e61cc1f1bea049e521be9a6c27daec9927c Oct 11 10:53:25.201567 master-1 kubenswrapper[4771]: I1011 10:53:25.201477 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" event={"ID":"37015f12-0983-4016-9f76-6d0e3f641f28","Type":"ContainerDied","Data":"9218883a276b672e1ae5ba8960bf79ca7ce273871d0ff647d43beb2d404896a3"} Oct 11 10:53:25.201747 master-1 kubenswrapper[4771]: I1011 10:53:25.201564 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-5b4bcc4d85-hc9b5" Oct 11 10:53:25.201747 master-1 kubenswrapper[4771]: I1011 10:53:25.201597 4771 scope.go:117] "RemoveContainer" containerID="b3e3b46fc901080e41385771d41f43fbd31766c94a2a5ea0b8a9cb3a8c03ad18" Oct 11 10:53:25.204119 master-1 kubenswrapper[4771]: I1011 10:53:25.204074 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/openstack-cell1-galera-1" event={"ID":"66c0cd85-28ea-42de-8432-8803026d3124","Type":"ContainerStarted","Data":"2240330aade2ce1379e6a3490bcdde650fa18d121d70f13d7932e07f7c1be46f"} Oct 11 10:53:25.205669 master-1 kubenswrapper[4771]: I1011 10:53:25.205619 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovsdbserver-sb-1" event={"ID":"c6e032cb-3993-4b10-afe4-7f77821c8583","Type":"ContainerStarted","Data":"f72206a2629114d93d81b6f349520e61cc1f1bea049e521be9a6c27daec9927c"} Oct 11 10:53:25.207852 master-1 kubenswrapper[4771]: I1011 10:53:25.207788 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovsdbserver-nb-1" event={"ID":"8f9b018c-eb14-4c27-adcb-ba613238c78b","Type":"ContainerStarted","Data":"c80aa209259f95b1296293e51f0616ef8ca222bf514bed45d9e3a7971dacd61a"} Oct 11 10:53:25.207852 master-1 kubenswrapper[4771]: I1011 10:53:25.207861 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovsdbserver-nb-1" event={"ID":"8f9b018c-eb14-4c27-adcb-ba613238c78b","Type":"ContainerStarted","Data":"38bed839fd575b2e9be016ec166012e515b5818caf5cbb15907f50b015b3bc84"} Oct 11 10:53:25.224084 master-1 kubenswrapper[4771]: I1011 10:53:25.222946 4771 scope.go:117] "RemoveContainer" containerID="6500f337e1810213b3c48514bdf7915497fff03dfebcbb66402d535cebd46613" Oct 11 10:53:25.236255 master-1 kubenswrapper[4771]: I1011 10:53:25.236189 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/openstack-cell1-galera-1" podStartSLOduration=32.236170963 podStartE2EDuration="32.236170963s" podCreationTimestamp="2025-10-11 10:52:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:53:25.232558128 +0000 UTC m=+1637.206784609" watchObservedRunningTime="2025-10-11 10:53:25.236170963 +0000 UTC m=+1637.210397414" Oct 11 10:53:25.263419 master-1 kubenswrapper[4771]: I1011 10:53:25.263182 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ovsdbserver-nb-1" podStartSLOduration=20.938178932 podStartE2EDuration="25.263150502s" podCreationTimestamp="2025-10-11 10:53:00 +0000 UTC" firstStartedPulling="2025-10-11 10:53:20.231672136 +0000 UTC m=+1632.205898577" lastFinishedPulling="2025-10-11 10:53:24.556643696 +0000 UTC m=+1636.530870147" observedRunningTime="2025-10-11 10:53:25.256672925 +0000 UTC m=+1637.230899366" watchObservedRunningTime="2025-10-11 10:53:25.263150502 +0000 UTC m=+1637.237376963" Oct 11 10:53:25.281029 master-1 kubenswrapper[4771]: I1011 10:53:25.279930 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-5b4bcc4d85-hc9b5"] Oct 11 10:53:25.285824 master-1 kubenswrapper[4771]: I1011 10:53:25.285766 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-5b4bcc4d85-hc9b5"] Oct 11 10:53:25.337881 master-1 kubenswrapper[4771]: I1011 10:53:25.337815 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:25.338004 master-1 kubenswrapper[4771]: I1011 10:53:25.337909 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:25.672760 master-1 kubenswrapper[4771]: I1011 10:53:25.672662 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:26.450018 master-1 kubenswrapper[4771]: I1011 10:53:26.449930 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37015f12-0983-4016-9f76-6d0e3f641f28" path="/var/lib/kubelet/pods/37015f12-0983-4016-9f76-6d0e3f641f28/volumes" Oct 11 10:53:27.193187 master-1 kubenswrapper[4771]: I1011 10:53:27.193107 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/openstack-galera-0" Oct 11 10:53:27.193187 master-1 kubenswrapper[4771]: I1011 10:53:27.193187 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/openstack-galera-0" Oct 11 10:53:27.243951 master-1 kubenswrapper[4771]: I1011 10:53:27.243836 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerStarted","Data":"1023c854292d2503211da52aaf16aa7e2199948c97ebed99bad537459ca3e33b"} Oct 11 10:53:27.254467 master-1 kubenswrapper[4771]: I1011 10:53:27.254401 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovsdbserver-sb-1" event={"ID":"c6e032cb-3993-4b10-afe4-7f77821c8583","Type":"ContainerStarted","Data":"bba61c0f4e00cda3da7bc8031cef1f99b6da7cf2a2f76cdd21295a1ccf9fbdbd"} Oct 11 10:53:27.254467 master-1 kubenswrapper[4771]: I1011 10:53:27.254475 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovsdbserver-sb-1" event={"ID":"c6e032cb-3993-4b10-afe4-7f77821c8583","Type":"ContainerStarted","Data":"a7d15c920a660899f780bf876eea2e54e7c7d2393a551aa2fbb8276e512a0a28"} Oct 11 10:53:27.337235 master-1 kubenswrapper[4771]: I1011 10:53:27.337121 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ovsdbserver-sb-1" podStartSLOduration=17.48348072 podStartE2EDuration="18.337090499s" podCreationTimestamp="2025-10-11 10:53:09 +0000 UTC" firstStartedPulling="2025-10-11 10:53:25.155531992 +0000 UTC m=+1637.129758433" lastFinishedPulling="2025-10-11 10:53:26.009141731 +0000 UTC m=+1637.983368212" observedRunningTime="2025-10-11 10:53:27.32677016 +0000 UTC m=+1639.300996601" watchObservedRunningTime="2025-10-11 10:53:27.337090499 +0000 UTC m=+1639.311316960" Oct 11 10:53:27.916215 master-1 kubenswrapper[4771]: I1011 10:53:27.916100 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:28.720023 master-1 kubenswrapper[4771]: I1011 10:53:28.719848 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:28.720739 master-1 kubenswrapper[4771]: I1011 10:53:28.720676 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:28.916848 master-1 kubenswrapper[4771]: I1011 10:53:28.916730 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:29.304195 master-1 kubenswrapper[4771]: I1011 10:53:29.304142 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ovsdbserver-nb-1" Oct 11 10:53:31.359173 master-1 kubenswrapper[4771]: I1011 10:53:31.359006 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-7b64bc6b99-wp674"] Oct 11 10:53:31.360142 master-1 kubenswrapper[4771]: E1011 10:53:31.360102 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="37015f12-0983-4016-9f76-6d0e3f641f28" containerName="init" Oct 11 10:53:31.360142 master-1 kubenswrapper[4771]: I1011 10:53:31.360136 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="37015f12-0983-4016-9f76-6d0e3f641f28" containerName="init" Oct 11 10:53:31.360230 master-1 kubenswrapper[4771]: E1011 10:53:31.360190 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="37015f12-0983-4016-9f76-6d0e3f641f28" containerName="dnsmasq-dns" Oct 11 10:53:31.360230 master-1 kubenswrapper[4771]: I1011 10:53:31.360201 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="37015f12-0983-4016-9f76-6d0e3f641f28" containerName="dnsmasq-dns" Oct 11 10:53:31.360556 master-1 kubenswrapper[4771]: I1011 10:53:31.360479 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="37015f12-0983-4016-9f76-6d0e3f641f28" containerName="dnsmasq-dns" Oct 11 10:53:31.365849 master-1 kubenswrapper[4771]: I1011 10:53:31.365780 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.370596 master-1 kubenswrapper[4771]: I1011 10:53:31.370087 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovsdbserver-nb" Oct 11 10:53:31.371469 master-1 kubenswrapper[4771]: I1011 10:53:31.370382 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovsdbserver-sb" Oct 11 10:53:31.414899 master-1 kubenswrapper[4771]: I1011 10:53:31.407710 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-7b64bc6b99-wp674"] Oct 11 10:53:31.414899 master-1 kubenswrapper[4771]: I1011 10:53:31.414434 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovn-northd-0"] Oct 11 10:53:31.416540 master-1 kubenswrapper[4771]: I1011 10:53:31.416504 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-northd-0" Oct 11 10:53:31.422968 master-1 kubenswrapper[4771]: I1011 10:53:31.422905 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovnnorthd-config" Oct 11 10:53:31.423071 master-1 kubenswrapper[4771]: I1011 10:53:31.422999 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ovnnorthd-ovndbs" Oct 11 10:53:31.423129 master-1 kubenswrapper[4771]: I1011 10:53:31.423043 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovnnorthd-scripts" Oct 11 10:53:31.423396 master-1 kubenswrapper[4771]: I1011 10:53:31.423343 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-northd-0"] Oct 11 10:53:31.441775 master-1 kubenswrapper[4771]: I1011 10:53:31.441616 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-config\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.441775 master-1 kubenswrapper[4771]: I1011 10:53:31.441717 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgbhb\" (UniqueName: \"kubernetes.io/projected/49a589aa-7d75-4aba-aca3-9fffa3d86378-kube-api-access-lgbhb\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.441775 master-1 kubenswrapper[4771]: I1011 10:53:31.441764 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-dns-svc\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.441961 master-1 kubenswrapper[4771]: I1011 10:53:31.441793 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-nb\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.441961 master-1 kubenswrapper[4771]: I1011 10:53:31.441827 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-sb\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.544103 master-1 kubenswrapper[4771]: I1011 10:53:31.544001 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-dns-svc\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.544103 master-1 kubenswrapper[4771]: I1011 10:53:31.544085 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-nb\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.544103 master-1 kubenswrapper[4771]: I1011 10:53:31.544120 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-rundir\" (UniqueName: \"kubernetes.io/empty-dir/e142309b-a5e4-48bc-913f-89bb35b61a51-ovn-rundir\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.544582 master-1 kubenswrapper[4771]: I1011 10:53:31.544143 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-sb\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.544582 master-1 kubenswrapper[4771]: I1011 10:53:31.544182 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/e142309b-a5e4-48bc-913f-89bb35b61a51-metrics-certs-tls-certs\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.544582 master-1 kubenswrapper[4771]: I1011 10:53:31.544231 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e142309b-a5e4-48bc-913f-89bb35b61a51-config\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.544582 master-1 kubenswrapper[4771]: I1011 10:53:31.544269 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp87r\" (UniqueName: \"kubernetes.io/projected/e142309b-a5e4-48bc-913f-89bb35b61a51-kube-api-access-lp87r\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.544582 master-1 kubenswrapper[4771]: I1011 10:53:31.544313 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/e142309b-a5e4-48bc-913f-89bb35b61a51-scripts\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.544582 master-1 kubenswrapper[4771]: I1011 10:53:31.544383 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e142309b-a5e4-48bc-913f-89bb35b61a51-combined-ca-bundle\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.544582 master-1 kubenswrapper[4771]: I1011 10:53:31.544417 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-config\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.544582 master-1 kubenswrapper[4771]: I1011 10:53:31.544445 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-northd-tls-certs\" (UniqueName: \"kubernetes.io/secret/e142309b-a5e4-48bc-913f-89bb35b61a51-ovn-northd-tls-certs\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.544582 master-1 kubenswrapper[4771]: I1011 10:53:31.544475 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbhb\" (UniqueName: \"kubernetes.io/projected/49a589aa-7d75-4aba-aca3-9fffa3d86378-kube-api-access-lgbhb\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.545693 master-1 kubenswrapper[4771]: I1011 10:53:31.545630 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-nb\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.545899 master-1 kubenswrapper[4771]: I1011 10:53:31.545803 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-dns-svc\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.545985 master-1 kubenswrapper[4771]: I1011 10:53:31.545922 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-config\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.546866 master-1 kubenswrapper[4771]: I1011 10:53:31.546827 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-sb\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.567394 master-1 kubenswrapper[4771]: I1011 10:53:31.567291 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbhb\" (UniqueName: \"kubernetes.io/projected/49a589aa-7d75-4aba-aca3-9fffa3d86378-kube-api-access-lgbhb\") pod \"dnsmasq-dns-7b64bc6b99-wp674\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.646106 master-1 kubenswrapper[4771]: I1011 10:53:31.646006 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/e142309b-a5e4-48bc-913f-89bb35b61a51-metrics-certs-tls-certs\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.646106 master-1 kubenswrapper[4771]: I1011 10:53:31.646103 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e142309b-a5e4-48bc-913f-89bb35b61a51-config\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.646596 master-1 kubenswrapper[4771]: I1011 10:53:31.646131 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-lp87r\" (UniqueName: \"kubernetes.io/projected/e142309b-a5e4-48bc-913f-89bb35b61a51-kube-api-access-lp87r\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.646596 master-1 kubenswrapper[4771]: I1011 10:53:31.646173 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/e142309b-a5e4-48bc-913f-89bb35b61a51-scripts\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.646596 master-1 kubenswrapper[4771]: I1011 10:53:31.646205 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e142309b-a5e4-48bc-913f-89bb35b61a51-combined-ca-bundle\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.646596 master-1 kubenswrapper[4771]: I1011 10:53:31.646240 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovn-northd-tls-certs\" (UniqueName: \"kubernetes.io/secret/e142309b-a5e4-48bc-913f-89bb35b61a51-ovn-northd-tls-certs\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.646596 master-1 kubenswrapper[4771]: I1011 10:53:31.646284 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovn-rundir\" (UniqueName: \"kubernetes.io/empty-dir/e142309b-a5e4-48bc-913f-89bb35b61a51-ovn-rundir\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.647461 master-1 kubenswrapper[4771]: I1011 10:53:31.647318 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovn-rundir\" (UniqueName: \"kubernetes.io/empty-dir/e142309b-a5e4-48bc-913f-89bb35b61a51-ovn-rundir\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.647994 master-1 kubenswrapper[4771]: I1011 10:53:31.647933 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/e142309b-a5e4-48bc-913f-89bb35b61a51-scripts\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.648054 master-1 kubenswrapper[4771]: I1011 10:53:31.647974 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e142309b-a5e4-48bc-913f-89bb35b61a51-config\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.651376 master-1 kubenswrapper[4771]: I1011 10:53:31.651309 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"metrics-certs-tls-certs\" (UniqueName: \"kubernetes.io/secret/e142309b-a5e4-48bc-913f-89bb35b61a51-metrics-certs-tls-certs\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.652333 master-1 kubenswrapper[4771]: I1011 10:53:31.652282 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovn-northd-tls-certs\" (UniqueName: \"kubernetes.io/secret/e142309b-a5e4-48bc-913f-89bb35b61a51-ovn-northd-tls-certs\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.652468 master-1 kubenswrapper[4771]: I1011 10:53:31.652395 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e142309b-a5e4-48bc-913f-89bb35b61a51-combined-ca-bundle\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.674275 master-1 kubenswrapper[4771]: I1011 10:53:31.674073 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp87r\" (UniqueName: \"kubernetes.io/projected/e142309b-a5e4-48bc-913f-89bb35b61a51-kube-api-access-lp87r\") pod \"ovn-northd-0\" (UID: \"e142309b-a5e4-48bc-913f-89bb35b61a51\") " pod="openstack/ovn-northd-0" Oct 11 10:53:31.724523 master-1 kubenswrapper[4771]: I1011 10:53:31.724415 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:31.738519 master-1 kubenswrapper[4771]: I1011 10:53:31.738235 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-northd-0" Oct 11 10:53:31.958347 master-1 kubenswrapper[4771]: I1011 10:53:31.958169 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:32.010379 master-1 kubenswrapper[4771]: I1011 10:53:32.010307 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ovsdbserver-sb-1" Oct 11 10:53:32.219408 master-1 kubenswrapper[4771]: W1011 10:53:32.218877 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode142309b_a5e4_48bc_913f_89bb35b61a51.slice/crio-1b05c8ea4ae87188bc6ea327174e59a0bd3129e474c9d9652dc8902c7ef5116f WatchSource:0}: Error finding container 1b05c8ea4ae87188bc6ea327174e59a0bd3129e474c9d9652dc8902c7ef5116f: Status 404 returned error can't find the container with id 1b05c8ea4ae87188bc6ea327174e59a0bd3129e474c9d9652dc8902c7ef5116f Oct 11 10:53:32.224496 master-1 kubenswrapper[4771]: I1011 10:53:32.224442 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-northd-0"] Oct 11 10:53:32.291236 master-1 kubenswrapper[4771]: I1011 10:53:32.291128 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-northd-0" event={"ID":"e142309b-a5e4-48bc-913f-89bb35b61a51","Type":"ContainerStarted","Data":"1b05c8ea4ae87188bc6ea327174e59a0bd3129e474c9d9652dc8902c7ef5116f"} Oct 11 10:53:32.311071 master-1 kubenswrapper[4771]: I1011 10:53:32.311006 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-7b64bc6b99-wp674"] Oct 11 10:53:32.312263 master-1 kubenswrapper[4771]: W1011 10:53:32.312185 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49a589aa_7d75_4aba_aca3_9fffa3d86378.slice/crio-e04ccf4542c4af977ce52340f83783b293bb99776af706005aa7ec0a114852af WatchSource:0}: Error finding container e04ccf4542c4af977ce52340f83783b293bb99776af706005aa7ec0a114852af: Status 404 returned error can't find the container with id e04ccf4542c4af977ce52340f83783b293bb99776af706005aa7ec0a114852af Oct 11 10:53:33.301185 master-1 kubenswrapper[4771]: I1011 10:53:33.301109 4771 generic.go:334] "Generic (PLEG): container finished" podID="49a589aa-7d75-4aba-aca3-9fffa3d86378" containerID="9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f" exitCode=0 Oct 11 10:53:33.301185 master-1 kubenswrapper[4771]: I1011 10:53:33.301175 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" event={"ID":"49a589aa-7d75-4aba-aca3-9fffa3d86378","Type":"ContainerDied","Data":"9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f"} Oct 11 10:53:33.301185 master-1 kubenswrapper[4771]: I1011 10:53:33.301205 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" event={"ID":"49a589aa-7d75-4aba-aca3-9fffa3d86378","Type":"ContainerStarted","Data":"e04ccf4542c4af977ce52340f83783b293bb99776af706005aa7ec0a114852af"} Oct 11 10:53:34.312439 master-1 kubenswrapper[4771]: I1011 10:53:34.312337 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-northd-0" event={"ID":"e142309b-a5e4-48bc-913f-89bb35b61a51","Type":"ContainerStarted","Data":"63206d81ced3a439d95ad40d612eb8d5678c748f255189d6f39d3bfe64481a95"} Oct 11 10:53:34.313069 master-1 kubenswrapper[4771]: I1011 10:53:34.312457 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-northd-0" event={"ID":"e142309b-a5e4-48bc-913f-89bb35b61a51","Type":"ContainerStarted","Data":"24f981b47ac3bf3f8c739d3b285b499930421c8e1dc44f02e79285a93ac186fa"} Oct 11 10:53:34.313069 master-1 kubenswrapper[4771]: I1011 10:53:34.312541 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ovn-northd-0" Oct 11 10:53:34.314914 master-1 kubenswrapper[4771]: I1011 10:53:34.314885 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" event={"ID":"49a589aa-7d75-4aba-aca3-9fffa3d86378","Type":"ContainerStarted","Data":"e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db"} Oct 11 10:53:34.315223 master-1 kubenswrapper[4771]: I1011 10:53:34.315160 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:34.620704 master-1 kubenswrapper[4771]: I1011 10:53:34.620571 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ovn-northd-0" podStartSLOduration=2.491795666 podStartE2EDuration="3.620537736s" podCreationTimestamp="2025-10-11 10:53:31 +0000 UTC" firstStartedPulling="2025-10-11 10:53:32.225303305 +0000 UTC m=+1644.199529746" lastFinishedPulling="2025-10-11 10:53:33.354045375 +0000 UTC m=+1645.328271816" observedRunningTime="2025-10-11 10:53:34.616918691 +0000 UTC m=+1646.591145152" watchObservedRunningTime="2025-10-11 10:53:34.620537736 +0000 UTC m=+1646.594764187" Oct 11 10:53:34.823833 master-1 kubenswrapper[4771]: I1011 10:53:34.823709 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" podStartSLOduration=3.823690777 podStartE2EDuration="3.823690777s" podCreationTimestamp="2025-10-11 10:53:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:53:34.821908205 +0000 UTC m=+1646.796134706" watchObservedRunningTime="2025-10-11 10:53:34.823690777 +0000 UTC m=+1646.797917228" Oct 11 10:53:36.336437 master-1 kubenswrapper[4771]: I1011 10:53:36.336343 4771 generic.go:334] "Generic (PLEG): container finished" podID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerID="1023c854292d2503211da52aaf16aa7e2199948c97ebed99bad537459ca3e33b" exitCode=0 Oct 11 10:53:36.337041 master-1 kubenswrapper[4771]: I1011 10:53:36.336459 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerDied","Data":"1023c854292d2503211da52aaf16aa7e2199948c97ebed99bad537459ca3e33b"} Oct 11 10:53:37.566315 master-1 kubenswrapper[4771]: I1011 10:53:37.566237 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/glance-db-create-dnsz5"] Oct 11 10:53:37.567678 master-1 kubenswrapper[4771]: I1011 10:53:37.567630 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-db-create-dnsz5" Oct 11 10:53:37.602344 master-1 kubenswrapper[4771]: I1011 10:53:37.602290 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-db-create-dnsz5"] Oct 11 10:53:37.695396 master-1 kubenswrapper[4771]: I1011 10:53:37.695268 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnbv\" (UniqueName: \"kubernetes.io/projected/57ac9130-d850-4420-a75e-53ec744b16eb-kube-api-access-2dnbv\") pod \"glance-db-create-dnsz5\" (UID: \"57ac9130-d850-4420-a75e-53ec744b16eb\") " pod="openstack/glance-db-create-dnsz5" Oct 11 10:53:37.797423 master-1 kubenswrapper[4771]: I1011 10:53:37.797340 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnbv\" (UniqueName: \"kubernetes.io/projected/57ac9130-d850-4420-a75e-53ec744b16eb-kube-api-access-2dnbv\") pod \"glance-db-create-dnsz5\" (UID: \"57ac9130-d850-4420-a75e-53ec744b16eb\") " pod="openstack/glance-db-create-dnsz5" Oct 11 10:53:37.804158 master-1 kubenswrapper[4771]: E1011 10:53:37.803993 4771 projected.go:194] Error preparing data for projected volume kube-api-access-2dnbv for pod openstack/glance-db-create-dnsz5: failed to fetch token: pod "glance-db-create-dnsz5" not found Oct 11 10:53:37.804158 master-1 kubenswrapper[4771]: E1011 10:53:37.804104 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ac9130-d850-4420-a75e-53ec744b16eb-kube-api-access-2dnbv podName:57ac9130-d850-4420-a75e-53ec744b16eb nodeName:}" failed. No retries permitted until 2025-10-11 10:53:38.304080708 +0000 UTC m=+1650.278307149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2dnbv" (UniqueName: "kubernetes.io/projected/57ac9130-d850-4420-a75e-53ec744b16eb-kube-api-access-2dnbv") pod "glance-db-create-dnsz5" (UID: "57ac9130-d850-4420-a75e-53ec744b16eb") : failed to fetch token: pod "glance-db-create-dnsz5" not found Oct 11 10:53:38.313685 master-1 kubenswrapper[4771]: I1011 10:53:38.312913 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnbv\" (UniqueName: \"kubernetes.io/projected/57ac9130-d850-4420-a75e-53ec744b16eb-kube-api-access-2dnbv\") pod \"glance-db-create-dnsz5\" (UID: \"57ac9130-d850-4420-a75e-53ec744b16eb\") " pod="openstack/glance-db-create-dnsz5" Oct 11 10:53:38.347149 master-1 kubenswrapper[4771]: I1011 10:53:38.346739 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnbv\" (UniqueName: \"kubernetes.io/projected/57ac9130-d850-4420-a75e-53ec744b16eb-kube-api-access-2dnbv\") pod \"glance-db-create-dnsz5\" (UID: \"57ac9130-d850-4420-a75e-53ec744b16eb\") " pod="openstack/glance-db-create-dnsz5" Oct 11 10:53:38.518916 master-1 kubenswrapper[4771]: I1011 10:53:38.518843 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-db-create-dnsz5" Oct 11 10:53:39.362222 master-1 kubenswrapper[4771]: I1011 10:53:39.362145 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-db-create-dnsz5"] Oct 11 10:53:39.373491 master-1 kubenswrapper[4771]: W1011 10:53:39.373411 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57ac9130_d850_4420_a75e_53ec744b16eb.slice/crio-2e1666340b01e4c1a48c00414db10d0677e3b3f3c913a32f6266f8c21f8e5f13 WatchSource:0}: Error finding container 2e1666340b01e4c1a48c00414db10d0677e3b3f3c913a32f6266f8c21f8e5f13: Status 404 returned error can't find the container with id 2e1666340b01e4c1a48c00414db10d0677e3b3f3c913a32f6266f8c21f8e5f13 Oct 11 10:53:40.372207 master-1 kubenswrapper[4771]: I1011 10:53:40.372131 4771 generic.go:334] "Generic (PLEG): container finished" podID="6fe99fba-e358-4203-a516-04b9ae19d789" containerID="7f8fc71d7ad02d8da77907079a53d04db1a0fb1212260a6e3e48d8f38e321946" exitCode=0 Oct 11 10:53:40.372924 master-1 kubenswrapper[4771]: I1011 10:53:40.372269 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/rabbitmq-server-0" event={"ID":"6fe99fba-e358-4203-a516-04b9ae19d789","Type":"ContainerDied","Data":"7f8fc71d7ad02d8da77907079a53d04db1a0fb1212260a6e3e48d8f38e321946"} Oct 11 10:53:40.374957 master-1 kubenswrapper[4771]: I1011 10:53:40.374915 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-db-create-dnsz5" event={"ID":"57ac9130-d850-4420-a75e-53ec744b16eb","Type":"ContainerStarted","Data":"497e015434504b4db642357797a1c623d7b35238dcc0952d89c6a79885be7010"} Oct 11 10:53:40.375012 master-1 kubenswrapper[4771]: I1011 10:53:40.374963 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-db-create-dnsz5" event={"ID":"57ac9130-d850-4420-a75e-53ec744b16eb","Type":"ContainerStarted","Data":"2e1666340b01e4c1a48c00414db10d0677e3b3f3c913a32f6266f8c21f8e5f13"} Oct 11 10:53:41.387134 master-1 kubenswrapper[4771]: I1011 10:53:41.387038 4771 generic.go:334] "Generic (PLEG): container finished" podID="831321b9-20ce-409b-8bdb-ec231aef5f35" containerID="46e81e63ab3ceec54c8e0da9448541aeaf71c73eb9783cb511b8ceaa6d4dbd06" exitCode=0 Oct 11 10:53:41.387874 master-1 kubenswrapper[4771]: I1011 10:53:41.387189 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/rabbitmq-cell1-server-0" event={"ID":"831321b9-20ce-409b-8bdb-ec231aef5f35","Type":"ContainerDied","Data":"46e81e63ab3ceec54c8e0da9448541aeaf71c73eb9783cb511b8ceaa6d4dbd06"} Oct 11 10:53:41.390136 master-1 kubenswrapper[4771]: I1011 10:53:41.390090 4771 generic.go:334] "Generic (PLEG): container finished" podID="57ac9130-d850-4420-a75e-53ec744b16eb" containerID="497e015434504b4db642357797a1c623d7b35238dcc0952d89c6a79885be7010" exitCode=0 Oct 11 10:53:41.390246 master-1 kubenswrapper[4771]: I1011 10:53:41.390148 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-db-create-dnsz5" event={"ID":"57ac9130-d850-4420-a75e-53ec744b16eb","Type":"ContainerDied","Data":"497e015434504b4db642357797a1c623d7b35238dcc0952d89c6a79885be7010"} Oct 11 10:53:41.490981 master-1 kubenswrapper[4771]: I1011 10:53:41.490884 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/ovn-controller-mtzk7" podUID="4ab25521-7fba-40c9-b3db-377b1d0ec7a1" containerName="ovn-controller" probeResult="failure" output=< Oct 11 10:53:41.490981 master-1 kubenswrapper[4771]: ERROR - ovn-controller connection status is 'not connected', expecting 'connected' status Oct 11 10:53:41.490981 master-1 kubenswrapper[4771]: > Oct 11 10:53:41.529430 master-1 kubenswrapper[4771]: I1011 10:53:41.523947 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:41.727551 master-1 kubenswrapper[4771]: I1011 10:53:41.727478 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:53:42.012814 master-1 kubenswrapper[4771]: I1011 10:53:42.012765 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-86d565bb9-85bsq"] Oct 11 10:53:42.013308 master-1 kubenswrapper[4771]: I1011 10:53:42.013273 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" podUID="0ae0f8e3-9e87-45b8-8313-a0b65cf33106" containerName="dnsmasq-dns" containerID="cri-o://14ed7d218f9217fbceb4436ac3f26fb55858bf77044f44bd18a2d4ffe4eacee3" gracePeriod=10 Oct 11 10:53:42.401509 master-1 kubenswrapper[4771]: I1011 10:53:42.401449 4771 generic.go:334] "Generic (PLEG): container finished" podID="0ae0f8e3-9e87-45b8-8313-a0b65cf33106" containerID="14ed7d218f9217fbceb4436ac3f26fb55858bf77044f44bd18a2d4ffe4eacee3" exitCode=0 Oct 11 10:53:42.402471 master-1 kubenswrapper[4771]: I1011 10:53:42.401533 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" event={"ID":"0ae0f8e3-9e87-45b8-8313-a0b65cf33106","Type":"ContainerDied","Data":"14ed7d218f9217fbceb4436ac3f26fb55858bf77044f44bd18a2d4ffe4eacee3"} Oct 11 10:53:42.404544 master-1 kubenswrapper[4771]: I1011 10:53:42.404435 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/rabbitmq-cell1-server-0" event={"ID":"831321b9-20ce-409b-8bdb-ec231aef5f35","Type":"ContainerStarted","Data":"2e859e6aac8242725e1925bb9f62522cc9495b0f285ec9272dc591ce52c18bcd"} Oct 11 10:53:42.405426 master-1 kubenswrapper[4771]: I1011 10:53:42.405400 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:53:42.407540 master-1 kubenswrapper[4771]: I1011 10:53:42.407508 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerStarted","Data":"dceeb58fe69a42771858429ecb1c63834e43aa17881864b8125d37688c790df5"} Oct 11 10:53:42.410874 master-1 kubenswrapper[4771]: I1011 10:53:42.410832 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/rabbitmq-server-0" event={"ID":"6fe99fba-e358-4203-a516-04b9ae19d789","Type":"ContainerStarted","Data":"deea6ef83eb5b94b2f3189eda972012964d768194580bfc4a7d45bbaf474b35e"} Oct 11 10:53:42.411740 master-1 kubenswrapper[4771]: I1011 10:53:42.411705 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/rabbitmq-server-0" Oct 11 10:53:42.475195 master-1 kubenswrapper[4771]: I1011 10:53:42.475109 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/rabbitmq-server-0" podStartSLOduration=47.558349019 podStartE2EDuration="52.475084508s" podCreationTimestamp="2025-10-11 10:52:50 +0000 UTC" firstStartedPulling="2025-10-11 10:53:01.408273851 +0000 UTC m=+1613.382500292" lastFinishedPulling="2025-10-11 10:53:06.3250093 +0000 UTC m=+1618.299235781" observedRunningTime="2025-10-11 10:53:42.473725728 +0000 UTC m=+1654.447952189" watchObservedRunningTime="2025-10-11 10:53:42.475084508 +0000 UTC m=+1654.449310949" Oct 11 10:53:42.475785 master-1 kubenswrapper[4771]: I1011 10:53:42.475742 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/rabbitmq-cell1-server-0" podStartSLOduration=51.475734236 podStartE2EDuration="51.475734236s" podCreationTimestamp="2025-10-11 10:52:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:53:42.438947163 +0000 UTC m=+1654.413173604" watchObservedRunningTime="2025-10-11 10:53:42.475734236 +0000 UTC m=+1654.449960687" Oct 11 10:53:42.599563 master-1 kubenswrapper[4771]: I1011 10:53:42.599497 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:42.710908 master-1 kubenswrapper[4771]: I1011 10:53:42.710653 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-dns-svc\") pod \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " Oct 11 10:53:42.710908 master-1 kubenswrapper[4771]: I1011 10:53:42.710885 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b4pv\" (UniqueName: \"kubernetes.io/projected/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-kube-api-access-7b4pv\") pod \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " Oct 11 10:53:42.712862 master-1 kubenswrapper[4771]: I1011 10:53:42.711151 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-config\") pod \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\" (UID: \"0ae0f8e3-9e87-45b8-8313-a0b65cf33106\") " Oct 11 10:53:42.715369 master-1 kubenswrapper[4771]: I1011 10:53:42.715155 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-kube-api-access-7b4pv" (OuterVolumeSpecName: "kube-api-access-7b4pv") pod "0ae0f8e3-9e87-45b8-8313-a0b65cf33106" (UID: "0ae0f8e3-9e87-45b8-8313-a0b65cf33106"). InnerVolumeSpecName "kube-api-access-7b4pv". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:53:42.759235 master-1 kubenswrapper[4771]: I1011 10:53:42.758101 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-config" (OuterVolumeSpecName: "config") pod "0ae0f8e3-9e87-45b8-8313-a0b65cf33106" (UID: "0ae0f8e3-9e87-45b8-8313-a0b65cf33106"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:53:42.767659 master-1 kubenswrapper[4771]: I1011 10:53:42.767590 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "0ae0f8e3-9e87-45b8-8313-a0b65cf33106" (UID: "0ae0f8e3-9e87-45b8-8313-a0b65cf33106"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:53:42.779767 master-1 kubenswrapper[4771]: I1011 10:53:42.779446 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-db-create-dnsz5" Oct 11 10:53:42.814452 master-1 kubenswrapper[4771]: I1011 10:53:42.812881 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dnbv\" (UniqueName: \"kubernetes.io/projected/57ac9130-d850-4420-a75e-53ec744b16eb-kube-api-access-2dnbv\") pod \"57ac9130-d850-4420-a75e-53ec744b16eb\" (UID: \"57ac9130-d850-4420-a75e-53ec744b16eb\") " Oct 11 10:53:42.814452 master-1 kubenswrapper[4771]: I1011 10:53:42.813438 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-7b4pv\" (UniqueName: \"kubernetes.io/projected/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-kube-api-access-7b4pv\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:42.814452 master-1 kubenswrapper[4771]: I1011 10:53:42.813459 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:42.814452 master-1 kubenswrapper[4771]: I1011 10:53:42.813473 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/0ae0f8e3-9e87-45b8-8313-a0b65cf33106-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:42.822206 master-1 kubenswrapper[4771]: I1011 10:53:42.822124 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ac9130-d850-4420-a75e-53ec744b16eb-kube-api-access-2dnbv" (OuterVolumeSpecName: "kube-api-access-2dnbv") pod "57ac9130-d850-4420-a75e-53ec744b16eb" (UID: "57ac9130-d850-4420-a75e-53ec744b16eb"). InnerVolumeSpecName "kube-api-access-2dnbv". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:53:42.914775 master-1 kubenswrapper[4771]: I1011 10:53:42.914710 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-2dnbv\" (UniqueName: \"kubernetes.io/projected/57ac9130-d850-4420-a75e-53ec744b16eb-kube-api-access-2dnbv\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:43.430003 master-1 kubenswrapper[4771]: I1011 10:53:43.429938 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-db-create-dnsz5" Oct 11 10:53:43.430003 master-1 kubenswrapper[4771]: I1011 10:53:43.429939 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-db-create-dnsz5" event={"ID":"57ac9130-d850-4420-a75e-53ec744b16eb","Type":"ContainerDied","Data":"2e1666340b01e4c1a48c00414db10d0677e3b3f3c913a32f6266f8c21f8e5f13"} Oct 11 10:53:43.430003 master-1 kubenswrapper[4771]: I1011 10:53:43.430011 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1666340b01e4c1a48c00414db10d0677e3b3f3c913a32f6266f8c21f8e5f13" Oct 11 10:53:43.436613 master-1 kubenswrapper[4771]: I1011 10:53:43.436553 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" Oct 11 10:53:43.437495 master-1 kubenswrapper[4771]: I1011 10:53:43.437441 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-86d565bb9-85bsq" event={"ID":"0ae0f8e3-9e87-45b8-8313-a0b65cf33106","Type":"ContainerDied","Data":"1af26e0c3df30b750253930a520b32ea24880275e960e13e9910257c86f202ff"} Oct 11 10:53:43.437570 master-1 kubenswrapper[4771]: I1011 10:53:43.437528 4771 scope.go:117] "RemoveContainer" containerID="14ed7d218f9217fbceb4436ac3f26fb55858bf77044f44bd18a2d4ffe4eacee3" Oct 11 10:53:43.494039 master-1 kubenswrapper[4771]: I1011 10:53:43.493986 4771 scope.go:117] "RemoveContainer" containerID="fd43d772f12b2955515b0207673e261220c08cfd99b72815c0e4dd5a30cfab8c" Oct 11 10:53:43.546059 master-1 kubenswrapper[4771]: I1011 10:53:43.545997 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-86d565bb9-85bsq"] Oct 11 10:53:43.555806 master-1 kubenswrapper[4771]: I1011 10:53:43.555717 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-86d565bb9-85bsq"] Oct 11 10:53:44.454432 master-1 kubenswrapper[4771]: I1011 10:53:44.454369 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ae0f8e3-9e87-45b8-8313-a0b65cf33106" path="/var/lib/kubelet/pods/0ae0f8e3-9e87-45b8-8313-a0b65cf33106/volumes" Oct 11 10:53:44.458423 master-1 kubenswrapper[4771]: I1011 10:53:44.458336 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerStarted","Data":"71476de09aca12c350c9bd4fa53bcc64aae8fab0a0998dc69e0421858e6b79c4"} Oct 11 10:53:46.518724 master-1 kubenswrapper[4771]: I1011 10:53:46.518568 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/ovn-controller-mtzk7" podUID="4ab25521-7fba-40c9-b3db-377b1d0ec7a1" containerName="ovn-controller" probeResult="failure" output=< Oct 11 10:53:46.518724 master-1 kubenswrapper[4771]: ERROR - ovn-controller connection status is 'not connected', expecting 'connected' status Oct 11 10:53:46.518724 master-1 kubenswrapper[4771]: > Oct 11 10:53:46.556738 master-1 kubenswrapper[4771]: I1011 10:53:46.556679 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ovn-controller-ovs-mvzxp" Oct 11 10:53:46.814052 master-1 kubenswrapper[4771]: I1011 10:53:46.813969 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ovn-northd-0" Oct 11 10:53:46.923935 master-1 kubenswrapper[4771]: I1011 10:53:46.923822 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovn-controller-mtzk7-config-gp4zd"] Oct 11 10:53:46.925197 master-1 kubenswrapper[4771]: E1011 10:53:46.924328 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="57ac9130-d850-4420-a75e-53ec744b16eb" containerName="mariadb-database-create" Oct 11 10:53:46.925197 master-1 kubenswrapper[4771]: I1011 10:53:46.924445 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ac9130-d850-4420-a75e-53ec744b16eb" containerName="mariadb-database-create" Oct 11 10:53:46.925197 master-1 kubenswrapper[4771]: E1011 10:53:46.924465 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0ae0f8e3-9e87-45b8-8313-a0b65cf33106" containerName="dnsmasq-dns" Oct 11 10:53:46.925197 master-1 kubenswrapper[4771]: I1011 10:53:46.924475 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae0f8e3-9e87-45b8-8313-a0b65cf33106" containerName="dnsmasq-dns" Oct 11 10:53:46.925197 master-1 kubenswrapper[4771]: E1011 10:53:46.924489 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0ae0f8e3-9e87-45b8-8313-a0b65cf33106" containerName="init" Oct 11 10:53:46.925197 master-1 kubenswrapper[4771]: I1011 10:53:46.924500 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae0f8e3-9e87-45b8-8313-a0b65cf33106" containerName="init" Oct 11 10:53:46.925197 master-1 kubenswrapper[4771]: I1011 10:53:46.924665 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="57ac9130-d850-4420-a75e-53ec744b16eb" containerName="mariadb-database-create" Oct 11 10:53:46.925197 master-1 kubenswrapper[4771]: I1011 10:53:46.924677 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0ae0f8e3-9e87-45b8-8313-a0b65cf33106" containerName="dnsmasq-dns" Oct 11 10:53:46.925492 master-1 kubenswrapper[4771]: I1011 10:53:46.925310 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:46.929003 master-1 kubenswrapper[4771]: I1011 10:53:46.928548 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovncontroller-extra-scripts" Oct 11 10:53:46.943705 master-1 kubenswrapper[4771]: I1011 10:53:46.943665 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-mtzk7-config-gp4zd"] Oct 11 10:53:47.106347 master-1 kubenswrapper[4771]: I1011 10:53:47.106213 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-additional-scripts\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.106347 master-1 kubenswrapper[4771]: I1011 10:53:47.106313 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run-ovn\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.106588 master-1 kubenswrapper[4771]: I1011 10:53:47.106346 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.106588 master-1 kubenswrapper[4771]: I1011 10:53:47.106410 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-log-ovn\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.106588 master-1 kubenswrapper[4771]: I1011 10:53:47.106437 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-scripts\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.106588 master-1 kubenswrapper[4771]: I1011 10:53:47.106497 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhn6l\" (UniqueName: \"kubernetes.io/projected/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-kube-api-access-nhn6l\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.209332 master-1 kubenswrapper[4771]: I1011 10:53:47.209227 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-log-ovn\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.209332 master-1 kubenswrapper[4771]: I1011 10:53:47.209322 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-scripts\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.209677 master-1 kubenswrapper[4771]: I1011 10:53:47.209441 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-nhn6l\" (UniqueName: \"kubernetes.io/projected/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-kube-api-access-nhn6l\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.209677 master-1 kubenswrapper[4771]: I1011 10:53:47.209534 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-log-ovn\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.209677 master-1 kubenswrapper[4771]: I1011 10:53:47.209588 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-additional-scripts\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.210833 master-1 kubenswrapper[4771]: I1011 10:53:47.209873 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run-ovn\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.210833 master-1 kubenswrapper[4771]: I1011 10:53:47.209922 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.210833 master-1 kubenswrapper[4771]: I1011 10:53:47.210215 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run-ovn\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.210833 master-1 kubenswrapper[4771]: I1011 10:53:47.210287 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.212399 master-1 kubenswrapper[4771]: I1011 10:53:47.211440 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-additional-scripts\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.214190 master-1 kubenswrapper[4771]: I1011 10:53:47.214114 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-scripts\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.235243 master-1 kubenswrapper[4771]: I1011 10:53:47.235084 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhn6l\" (UniqueName: \"kubernetes.io/projected/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-kube-api-access-nhn6l\") pod \"ovn-controller-mtzk7-config-gp4zd\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.321015 master-1 kubenswrapper[4771]: I1011 10:53:47.320938 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:47.904246 master-1 kubenswrapper[4771]: I1011 10:53:47.904135 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/openstack-galera-0" Oct 11 10:53:47.938426 master-1 kubenswrapper[4771]: I1011 10:53:47.938276 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:47.972788 master-1 kubenswrapper[4771]: I1011 10:53:47.972726 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/openstack-galera-0" Oct 11 10:53:48.010538 master-1 kubenswrapper[4771]: I1011 10:53:48.010315 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/openstack-cell1-galera-1" Oct 11 10:53:48.146851 master-1 kubenswrapper[4771]: I1011 10:53:48.146781 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-mtzk7-config-gp4zd"] Oct 11 10:53:48.154509 master-1 kubenswrapper[4771]: W1011 10:53:48.154460 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf12bbc67_6a5d_4c1c_a685_c26c2b70a0a1.slice/crio-d2ed419b04ee231b2641d62eb1fa5b9efe937fb234dd74ec625c225a1769b347 WatchSource:0}: Error finding container d2ed419b04ee231b2641d62eb1fa5b9efe937fb234dd74ec625c225a1769b347: Status 404 returned error can't find the container with id d2ed419b04ee231b2641d62eb1fa5b9efe937fb234dd74ec625c225a1769b347 Oct 11 10:53:48.509106 master-1 kubenswrapper[4771]: I1011 10:53:48.509020 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7-config-gp4zd" event={"ID":"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1","Type":"ContainerStarted","Data":"d4634f70346f96ae4f97fe711847f0e072862de8b631ac6a0aaa341026f8675e"} Oct 11 10:53:48.509509 master-1 kubenswrapper[4771]: I1011 10:53:48.509491 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7-config-gp4zd" event={"ID":"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1","Type":"ContainerStarted","Data":"d2ed419b04ee231b2641d62eb1fa5b9efe937fb234dd74ec625c225a1769b347"} Oct 11 10:53:48.513201 master-1 kubenswrapper[4771]: I1011 10:53:48.513100 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerStarted","Data":"92f430c78bc4b24a68001349b3a9ac48c77542314f208344451dc9fc116683d7"} Oct 11 10:53:48.555949 master-1 kubenswrapper[4771]: I1011 10:53:48.555867 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ovn-controller-mtzk7-config-gp4zd" podStartSLOduration=2.555845437 podStartE2EDuration="2.555845437s" podCreationTimestamp="2025-10-11 10:53:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:53:48.552683236 +0000 UTC m=+1660.526909757" watchObservedRunningTime="2025-10-11 10:53:48.555845437 +0000 UTC m=+1660.530071878" Oct 11 10:53:48.589557 master-1 kubenswrapper[4771]: I1011 10:53:48.589473 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/prometheus-metric-storage-0" podStartSLOduration=22.308501306 podStartE2EDuration="51.589453208s" podCreationTimestamp="2025-10-11 10:52:57 +0000 UTC" firstStartedPulling="2025-10-11 10:53:18.448247126 +0000 UTC m=+1630.422473567" lastFinishedPulling="2025-10-11 10:53:47.729198998 +0000 UTC m=+1659.703425469" observedRunningTime="2025-10-11 10:53:48.588350117 +0000 UTC m=+1660.562576648" watchObservedRunningTime="2025-10-11 10:53:48.589453208 +0000 UTC m=+1660.563679649" Oct 11 10:53:49.522287 master-1 kubenswrapper[4771]: I1011 10:53:49.522194 4771 generic.go:334] "Generic (PLEG): container finished" podID="f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" containerID="d4634f70346f96ae4f97fe711847f0e072862de8b631ac6a0aaa341026f8675e" exitCode=0 Oct 11 10:53:49.524867 master-1 kubenswrapper[4771]: I1011 10:53:49.524819 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7-config-gp4zd" event={"ID":"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1","Type":"ContainerDied","Data":"d4634f70346f96ae4f97fe711847f0e072862de8b631ac6a0aaa341026f8675e"} Oct 11 10:53:51.039494 master-1 kubenswrapper[4771]: I1011 10:53:51.039439 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:51.199976 master-1 kubenswrapper[4771]: I1011 10:53:51.199746 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-additional-scripts\") pod \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " Oct 11 10:53:51.199976 master-1 kubenswrapper[4771]: I1011 10:53:51.199817 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhn6l\" (UniqueName: \"kubernetes.io/projected/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-kube-api-access-nhn6l\") pod \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " Oct 11 10:53:51.199976 master-1 kubenswrapper[4771]: I1011 10:53:51.199850 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-scripts\") pod \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " Oct 11 10:53:51.199976 master-1 kubenswrapper[4771]: I1011 10:53:51.199924 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run-ovn\") pod \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " Oct 11 10:53:51.201004 master-1 kubenswrapper[4771]: I1011 10:53:51.200022 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-log-ovn\") pod \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " Oct 11 10:53:51.201004 master-1 kubenswrapper[4771]: I1011 10:53:51.200047 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run\") pod \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\" (UID: \"f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1\") " Oct 11 10:53:51.201004 master-1 kubenswrapper[4771]: I1011 10:53:51.200467 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run" (OuterVolumeSpecName: "var-run") pod "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" (UID: "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1"). InnerVolumeSpecName "var-run". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:53:51.201004 master-1 kubenswrapper[4771]: I1011 10:53:51.200625 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-additional-scripts" (OuterVolumeSpecName: "additional-scripts") pod "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" (UID: "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1"). InnerVolumeSpecName "additional-scripts". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:53:51.201004 master-1 kubenswrapper[4771]: I1011 10:53:51.200705 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run-ovn" (OuterVolumeSpecName: "var-run-ovn") pod "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" (UID: "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1"). InnerVolumeSpecName "var-run-ovn". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:53:51.201004 master-1 kubenswrapper[4771]: I1011 10:53:51.200719 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-log-ovn" (OuterVolumeSpecName: "var-log-ovn") pod "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" (UID: "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1"). InnerVolumeSpecName "var-log-ovn". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:53:51.201487 master-1 kubenswrapper[4771]: I1011 10:53:51.201304 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-scripts" (OuterVolumeSpecName: "scripts") pod "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" (UID: "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:53:51.204581 master-1 kubenswrapper[4771]: I1011 10:53:51.203774 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-kube-api-access-nhn6l" (OuterVolumeSpecName: "kube-api-access-nhn6l") pod "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" (UID: "f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1"). InnerVolumeSpecName "kube-api-access-nhn6l". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:53:51.302334 master-1 kubenswrapper[4771]: I1011 10:53:51.302238 4771 reconciler_common.go:293] "Volume detached for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run-ovn\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:51.302334 master-1 kubenswrapper[4771]: I1011 10:53:51.302298 4771 reconciler_common.go:293] "Volume detached for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-log-ovn\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:51.302334 master-1 kubenswrapper[4771]: I1011 10:53:51.302325 4771 reconciler_common.go:293] "Volume detached for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-var-run\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:51.302334 master-1 kubenswrapper[4771]: I1011 10:53:51.302342 4771 reconciler_common.go:293] "Volume detached for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-additional-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:51.302334 master-1 kubenswrapper[4771]: I1011 10:53:51.302389 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-nhn6l\" (UniqueName: \"kubernetes.io/projected/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-kube-api-access-nhn6l\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:51.302864 master-1 kubenswrapper[4771]: I1011 10:53:51.302405 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:51.348955 master-1 kubenswrapper[4771]: I1011 10:53:51.348854 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ovn-controller-mtzk7-config-gp4zd"] Oct 11 10:53:51.357424 master-1 kubenswrapper[4771]: I1011 10:53:51.357328 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ovn-controller-mtzk7-config-gp4zd"] Oct 11 10:53:51.515237 master-1 kubenswrapper[4771]: I1011 10:53:51.510627 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ovn-controller-mtzk7" Oct 11 10:53:51.556548 master-1 kubenswrapper[4771]: I1011 10:53:51.556475 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ed419b04ee231b2641d62eb1fa5b9efe937fb234dd74ec625c225a1769b347" Oct 11 10:53:51.556871 master-1 kubenswrapper[4771]: I1011 10:53:51.556594 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-gp4zd" Oct 11 10:53:51.638883 master-1 kubenswrapper[4771]: I1011 10:53:51.638805 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovn-controller-mtzk7-config-9fxhw"] Oct 11 10:53:51.640003 master-1 kubenswrapper[4771]: E1011 10:53:51.639970 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" containerName="ovn-config" Oct 11 10:53:51.640409 master-1 kubenswrapper[4771]: I1011 10:53:51.640341 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" containerName="ovn-config" Oct 11 10:53:51.640881 master-1 kubenswrapper[4771]: I1011 10:53:51.640854 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" containerName="ovn-config" Oct 11 10:53:51.642209 master-1 kubenswrapper[4771]: I1011 10:53:51.642176 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.646912 master-1 kubenswrapper[4771]: I1011 10:53:51.646876 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovncontroller-extra-scripts" Oct 11 10:53:51.660501 master-1 kubenswrapper[4771]: I1011 10:53:51.659556 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-mtzk7-config-9fxhw"] Oct 11 10:53:51.839475 master-1 kubenswrapper[4771]: I1011 10:53:51.839326 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqprx\" (UniqueName: \"kubernetes.io/projected/c637296f-4521-4528-b3f3-e247deac1ad8-kube-api-access-zqprx\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.840320 master-1 kubenswrapper[4771]: I1011 10:53:51.840248 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-log-ovn\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.840535 master-1 kubenswrapper[4771]: I1011 10:53:51.840483 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-scripts\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.840624 master-1 kubenswrapper[4771]: I1011 10:53:51.840572 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.840700 master-1 kubenswrapper[4771]: I1011 10:53:51.840643 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run-ovn\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.840913 master-1 kubenswrapper[4771]: I1011 10:53:51.840854 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-additional-scripts\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.942894 master-1 kubenswrapper[4771]: I1011 10:53:51.942782 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-zqprx\" (UniqueName: \"kubernetes.io/projected/c637296f-4521-4528-b3f3-e247deac1ad8-kube-api-access-zqprx\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.942894 master-1 kubenswrapper[4771]: I1011 10:53:51.942888 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-log-ovn\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.943457 master-1 kubenswrapper[4771]: I1011 10:53:51.942991 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-scripts\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.943457 master-1 kubenswrapper[4771]: I1011 10:53:51.943024 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.943457 master-1 kubenswrapper[4771]: I1011 10:53:51.943065 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run-ovn\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.943457 master-1 kubenswrapper[4771]: I1011 10:53:51.943116 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-additional-scripts\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.943457 master-1 kubenswrapper[4771]: I1011 10:53:51.943181 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-log-ovn\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.943457 master-1 kubenswrapper[4771]: I1011 10:53:51.943348 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run-ovn\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.943457 master-1 kubenswrapper[4771]: I1011 10:53:51.943403 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.944826 master-1 kubenswrapper[4771]: I1011 10:53:51.944774 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-additional-scripts\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.947752 master-1 kubenswrapper[4771]: I1011 10:53:51.947671 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-scripts\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.970455 master-1 kubenswrapper[4771]: I1011 10:53:51.969257 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqprx\" (UniqueName: \"kubernetes.io/projected/c637296f-4521-4528-b3f3-e247deac1ad8-kube-api-access-zqprx\") pod \"ovn-controller-mtzk7-config-9fxhw\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:51.971738 master-1 kubenswrapper[4771]: I1011 10:53:51.971692 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:52.426417 master-1 kubenswrapper[4771]: I1011 10:53:52.426312 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-mtzk7-config-9fxhw"] Oct 11 10:53:52.431311 master-1 kubenswrapper[4771]: W1011 10:53:52.431248 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc637296f_4521_4528_b3f3_e247deac1ad8.slice/crio-85ec1a1327a46e2b03fec9261ebaecc8a7d06803923507fa15cf1db0337a471c WatchSource:0}: Error finding container 85ec1a1327a46e2b03fec9261ebaecc8a7d06803923507fa15cf1db0337a471c: Status 404 returned error can't find the container with id 85ec1a1327a46e2b03fec9261ebaecc8a7d06803923507fa15cf1db0337a471c Oct 11 10:53:52.455685 master-1 kubenswrapper[4771]: I1011 10:53:52.455574 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1" path="/var/lib/kubelet/pods/f12bbc67-6a5d-4c1c-a685-c26c2b70a0a1/volumes" Oct 11 10:53:52.570835 master-1 kubenswrapper[4771]: I1011 10:53:52.570705 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7-config-9fxhw" event={"ID":"c637296f-4521-4528-b3f3-e247deac1ad8","Type":"ContainerStarted","Data":"85ec1a1327a46e2b03fec9261ebaecc8a7d06803923507fa15cf1db0337a471c"} Oct 11 10:53:52.622904 master-1 kubenswrapper[4771]: I1011 10:53:52.622845 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-6c99f4877f-vjhdt"] Oct 11 10:53:52.624886 master-1 kubenswrapper[4771]: I1011 10:53:52.624830 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.652315 master-1 kubenswrapper[4771]: I1011 10:53:52.646320 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-6c99f4877f-vjhdt"] Oct 11 10:53:52.663217 master-1 kubenswrapper[4771]: I1011 10:53:52.663152 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-sb\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.663473 master-1 kubenswrapper[4771]: I1011 10:53:52.663240 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-dns-svc\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.663541 master-1 kubenswrapper[4771]: I1011 10:53:52.663447 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-nb\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.663755 master-1 kubenswrapper[4771]: I1011 10:53:52.663706 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-config\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.663811 master-1 kubenswrapper[4771]: I1011 10:53:52.663774 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m5q5\" (UniqueName: \"kubernetes.io/projected/30700706-219b-47c1-83cd-278584a3f182-kube-api-access-2m5q5\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.765813 master-1 kubenswrapper[4771]: I1011 10:53:52.765738 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-config\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.765813 master-1 kubenswrapper[4771]: I1011 10:53:52.765790 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-2m5q5\" (UniqueName: \"kubernetes.io/projected/30700706-219b-47c1-83cd-278584a3f182-kube-api-access-2m5q5\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.766021 master-1 kubenswrapper[4771]: I1011 10:53:52.765844 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-sb\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.766021 master-1 kubenswrapper[4771]: I1011 10:53:52.765872 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-dns-svc\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.766021 master-1 kubenswrapper[4771]: I1011 10:53:52.765897 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-nb\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.767141 master-1 kubenswrapper[4771]: I1011 10:53:52.767087 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-config\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.768837 master-1 kubenswrapper[4771]: I1011 10:53:52.768776 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-dns-svc\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.771411 master-1 kubenswrapper[4771]: I1011 10:53:52.771331 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-nb\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.780168 master-1 kubenswrapper[4771]: I1011 10:53:52.780093 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-sb\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.791792 master-1 kubenswrapper[4771]: I1011 10:53:52.791714 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m5q5\" (UniqueName: \"kubernetes.io/projected/30700706-219b-47c1-83cd-278584a3f182-kube-api-access-2m5q5\") pod \"dnsmasq-dns-6c99f4877f-vjhdt\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:52.898382 master-1 kubenswrapper[4771]: I1011 10:53:52.898314 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/prometheus-metric-storage-0" Oct 11 10:53:52.947449 master-1 kubenswrapper[4771]: I1011 10:53:52.947386 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:53.396994 master-1 kubenswrapper[4771]: I1011 10:53:53.396924 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-6c99f4877f-vjhdt"] Oct 11 10:53:53.399851 master-1 kubenswrapper[4771]: W1011 10:53:53.399730 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30700706_219b_47c1_83cd_278584a3f182.slice/crio-cc8ace77e3138b8e3e45d04fc9090a4ecc543f54342ec2f309cdbc89855a76b5 WatchSource:0}: Error finding container cc8ace77e3138b8e3e45d04fc9090a4ecc543f54342ec2f309cdbc89855a76b5: Status 404 returned error can't find the container with id cc8ace77e3138b8e3e45d04fc9090a4ecc543f54342ec2f309cdbc89855a76b5 Oct 11 10:53:53.589159 master-1 kubenswrapper[4771]: I1011 10:53:53.589050 4771 generic.go:334] "Generic (PLEG): container finished" podID="c637296f-4521-4528-b3f3-e247deac1ad8" containerID="94fe8e005fb0a8b586a5c6a1e344905a51e3390259171a77f131bc97d101f438" exitCode=0 Oct 11 10:53:53.590079 master-1 kubenswrapper[4771]: I1011 10:53:53.589560 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7-config-9fxhw" event={"ID":"c637296f-4521-4528-b3f3-e247deac1ad8","Type":"ContainerDied","Data":"94fe8e005fb0a8b586a5c6a1e344905a51e3390259171a77f131bc97d101f438"} Oct 11 10:53:53.592309 master-1 kubenswrapper[4771]: I1011 10:53:53.592259 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" event={"ID":"30700706-219b-47c1-83cd-278584a3f182","Type":"ContainerStarted","Data":"cc8ace77e3138b8e3e45d04fc9090a4ecc543f54342ec2f309cdbc89855a76b5"} Oct 11 10:53:54.444836 master-1 kubenswrapper[4771]: I1011 10:53:54.444724 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/rabbitmq-cell1-server-0" podUID="831321b9-20ce-409b-8bdb-ec231aef5f35" containerName="rabbitmq" probeResult="failure" output="dial tcp 10.129.0.107:5671: connect: connection refused" Oct 11 10:53:54.605725 master-1 kubenswrapper[4771]: I1011 10:53:54.605636 4771 generic.go:334] "Generic (PLEG): container finished" podID="30700706-219b-47c1-83cd-278584a3f182" containerID="06ae06abec101801ffcb11de5d066d694be4874cbd2110b56c80026a91417fc8" exitCode=0 Oct 11 10:53:54.606656 master-1 kubenswrapper[4771]: I1011 10:53:54.605801 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" event={"ID":"30700706-219b-47c1-83cd-278584a3f182","Type":"ContainerDied","Data":"06ae06abec101801ffcb11de5d066d694be4874cbd2110b56c80026a91417fc8"} Oct 11 10:53:55.057533 master-1 kubenswrapper[4771]: I1011 10:53:55.057486 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:55.213089 master-1 kubenswrapper[4771]: I1011 10:53:55.212983 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqprx\" (UniqueName: \"kubernetes.io/projected/c637296f-4521-4528-b3f3-e247deac1ad8-kube-api-access-zqprx\") pod \"c637296f-4521-4528-b3f3-e247deac1ad8\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " Oct 11 10:53:55.213411 master-1 kubenswrapper[4771]: I1011 10:53:55.213120 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-scripts\") pod \"c637296f-4521-4528-b3f3-e247deac1ad8\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " Oct 11 10:53:55.213655 master-1 kubenswrapper[4771]: I1011 10:53:55.213612 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-log-ovn\") pod \"c637296f-4521-4528-b3f3-e247deac1ad8\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " Oct 11 10:53:55.213755 master-1 kubenswrapper[4771]: I1011 10:53:55.213667 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-additional-scripts\") pod \"c637296f-4521-4528-b3f3-e247deac1ad8\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " Oct 11 10:53:55.213755 master-1 kubenswrapper[4771]: I1011 10:53:55.213732 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run-ovn\") pod \"c637296f-4521-4528-b3f3-e247deac1ad8\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " Oct 11 10:53:55.213890 master-1 kubenswrapper[4771]: I1011 10:53:55.213766 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run\") pod \"c637296f-4521-4528-b3f3-e247deac1ad8\" (UID: \"c637296f-4521-4528-b3f3-e247deac1ad8\") " Oct 11 10:53:55.213890 master-1 kubenswrapper[4771]: I1011 10:53:55.213818 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-log-ovn" (OuterVolumeSpecName: "var-log-ovn") pod "c637296f-4521-4528-b3f3-e247deac1ad8" (UID: "c637296f-4521-4528-b3f3-e247deac1ad8"). InnerVolumeSpecName "var-log-ovn". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:53:55.214018 master-1 kubenswrapper[4771]: I1011 10:53:55.213905 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run-ovn" (OuterVolumeSpecName: "var-run-ovn") pod "c637296f-4521-4528-b3f3-e247deac1ad8" (UID: "c637296f-4521-4528-b3f3-e247deac1ad8"). InnerVolumeSpecName "var-run-ovn". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:53:55.214018 master-1 kubenswrapper[4771]: I1011 10:53:55.213931 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run" (OuterVolumeSpecName: "var-run") pod "c637296f-4521-4528-b3f3-e247deac1ad8" (UID: "c637296f-4521-4528-b3f3-e247deac1ad8"). InnerVolumeSpecName "var-run". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:53:55.215215 master-1 kubenswrapper[4771]: I1011 10:53:55.215164 4771 reconciler_common.go:293] "Volume detached for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-log-ovn\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:55.215323 master-1 kubenswrapper[4771]: I1011 10:53:55.215217 4771 reconciler_common.go:293] "Volume detached for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run-ovn\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:55.215432 master-1 kubenswrapper[4771]: I1011 10:53:55.215349 4771 reconciler_common.go:293] "Volume detached for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/c637296f-4521-4528-b3f3-e247deac1ad8-var-run\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:55.215432 master-1 kubenswrapper[4771]: I1011 10:53:55.215145 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-additional-scripts" (OuterVolumeSpecName: "additional-scripts") pod "c637296f-4521-4528-b3f3-e247deac1ad8" (UID: "c637296f-4521-4528-b3f3-e247deac1ad8"). InnerVolumeSpecName "additional-scripts". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:53:55.215432 master-1 kubenswrapper[4771]: I1011 10:53:55.215272 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-scripts" (OuterVolumeSpecName: "scripts") pod "c637296f-4521-4528-b3f3-e247deac1ad8" (UID: "c637296f-4521-4528-b3f3-e247deac1ad8"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:53:55.218323 master-1 kubenswrapper[4771]: I1011 10:53:55.218253 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c637296f-4521-4528-b3f3-e247deac1ad8-kube-api-access-zqprx" (OuterVolumeSpecName: "kube-api-access-zqprx") pod "c637296f-4521-4528-b3f3-e247deac1ad8" (UID: "c637296f-4521-4528-b3f3-e247deac1ad8"). InnerVolumeSpecName "kube-api-access-zqprx". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:53:55.317274 master-1 kubenswrapper[4771]: I1011 10:53:55.317161 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-zqprx\" (UniqueName: \"kubernetes.io/projected/c637296f-4521-4528-b3f3-e247deac1ad8-kube-api-access-zqprx\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:55.317274 master-1 kubenswrapper[4771]: I1011 10:53:55.317237 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:55.317274 master-1 kubenswrapper[4771]: I1011 10:53:55.317258 4771 reconciler_common.go:293] "Volume detached for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/c637296f-4521-4528-b3f3-e247deac1ad8-additional-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:53:55.621516 master-1 kubenswrapper[4771]: I1011 10:53:55.620053 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" event={"ID":"30700706-219b-47c1-83cd-278584a3f182","Type":"ContainerStarted","Data":"2dcdb27cf0dbce506998b4c8cfe73f7847cd892689b076fcba313e976b8a5349"} Oct 11 10:53:55.621516 master-1 kubenswrapper[4771]: I1011 10:53:55.620328 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:53:55.622195 master-1 kubenswrapper[4771]: I1011 10:53:55.622179 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7-config-9fxhw" event={"ID":"c637296f-4521-4528-b3f3-e247deac1ad8","Type":"ContainerDied","Data":"85ec1a1327a46e2b03fec9261ebaecc8a7d06803923507fa15cf1db0337a471c"} Oct 11 10:53:55.622250 master-1 kubenswrapper[4771]: I1011 10:53:55.622205 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85ec1a1327a46e2b03fec9261ebaecc8a7d06803923507fa15cf1db0337a471c" Oct 11 10:53:55.622325 master-1 kubenswrapper[4771]: I1011 10:53:55.622290 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-9fxhw" Oct 11 10:53:55.665393 master-1 kubenswrapper[4771]: I1011 10:53:55.665269 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" podStartSLOduration=3.665236964 podStartE2EDuration="3.665236964s" podCreationTimestamp="2025-10-11 10:53:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:53:55.660325332 +0000 UTC m=+1667.634551793" watchObservedRunningTime="2025-10-11 10:53:55.665236964 +0000 UTC m=+1667.639463405" Oct 11 10:53:56.184182 master-1 kubenswrapper[4771]: I1011 10:53:56.184089 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ovn-controller-mtzk7-config-9fxhw"] Oct 11 10:53:56.191213 master-1 kubenswrapper[4771]: I1011 10:53:56.191135 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ovn-controller-mtzk7-config-9fxhw"] Oct 11 10:53:56.360400 master-1 kubenswrapper[4771]: I1011 10:53:56.360275 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovn-controller-mtzk7-config-292gb"] Oct 11 10:53:56.360850 master-1 kubenswrapper[4771]: E1011 10:53:56.360814 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="c637296f-4521-4528-b3f3-e247deac1ad8" containerName="ovn-config" Oct 11 10:53:56.360850 master-1 kubenswrapper[4771]: I1011 10:53:56.360837 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="c637296f-4521-4528-b3f3-e247deac1ad8" containerName="ovn-config" Oct 11 10:53:56.361103 master-1 kubenswrapper[4771]: I1011 10:53:56.361075 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="c637296f-4521-4528-b3f3-e247deac1ad8" containerName="ovn-config" Oct 11 10:53:56.363428 master-1 kubenswrapper[4771]: I1011 10:53:56.362100 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.367110 master-1 kubenswrapper[4771]: I1011 10:53:56.366833 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovncontroller-extra-scripts" Oct 11 10:53:56.372178 master-1 kubenswrapper[4771]: I1011 10:53:56.369186 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-mtzk7-config-292gb"] Oct 11 10:53:56.446073 master-1 kubenswrapper[4771]: I1011 10:53:56.445937 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c637296f-4521-4528-b3f3-e247deac1ad8" path="/var/lib/kubelet/pods/c637296f-4521-4528-b3f3-e247deac1ad8/volumes" Oct 11 10:53:56.548048 master-1 kubenswrapper[4771]: I1011 10:53:56.547976 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run-ovn\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.548048 master-1 kubenswrapper[4771]: I1011 10:53:56.548048 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8cm\" (UniqueName: \"kubernetes.io/projected/5a75aab2-123f-491f-af07-939ade33aadc-kube-api-access-jg8cm\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.548236 master-1 kubenswrapper[4771]: I1011 10:53:56.548104 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-additional-scripts\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.548236 master-1 kubenswrapper[4771]: I1011 10:53:56.548207 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-log-ovn\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.548328 master-1 kubenswrapper[4771]: I1011 10:53:56.548259 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.548328 master-1 kubenswrapper[4771]: I1011 10:53:56.548298 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-scripts\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.650897 master-1 kubenswrapper[4771]: I1011 10:53:56.650279 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-additional-scripts\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.650897 master-1 kubenswrapper[4771]: I1011 10:53:56.650401 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-log-ovn\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.650897 master-1 kubenswrapper[4771]: I1011 10:53:56.650443 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.650897 master-1 kubenswrapper[4771]: I1011 10:53:56.650480 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-scripts\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.650897 master-1 kubenswrapper[4771]: I1011 10:53:56.650541 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run-ovn\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.650897 master-1 kubenswrapper[4771]: I1011 10:53:56.650567 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8cm\" (UniqueName: \"kubernetes.io/projected/5a75aab2-123f-491f-af07-939ade33aadc-kube-api-access-jg8cm\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.652082 master-1 kubenswrapper[4771]: I1011 10:53:56.651831 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-additional-scripts\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.652082 master-1 kubenswrapper[4771]: I1011 10:53:56.651982 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.652225 master-1 kubenswrapper[4771]: I1011 10:53:56.652093 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-log-ovn\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.652225 master-1 kubenswrapper[4771]: I1011 10:53:56.652157 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run-ovn\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.653957 master-1 kubenswrapper[4771]: I1011 10:53:56.653902 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-scripts\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.767321 master-1 kubenswrapper[4771]: I1011 10:53:56.766318 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8cm\" (UniqueName: \"kubernetes.io/projected/5a75aab2-123f-491f-af07-939ade33aadc-kube-api-access-jg8cm\") pod \"ovn-controller-mtzk7-config-292gb\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:56.990754 master-1 kubenswrapper[4771]: I1011 10:53:56.990546 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:53:57.011757 master-1 kubenswrapper[4771]: I1011 10:53:57.011652 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/rabbitmq-server-0" podUID="6fe99fba-e358-4203-a516-04b9ae19d789" containerName="rabbitmq" probeResult="failure" output="dial tcp 10.129.0.103:5671: connect: connection refused" Oct 11 10:53:57.482169 master-1 kubenswrapper[4771]: I1011 10:53:57.482118 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-controller-mtzk7-config-292gb"] Oct 11 10:53:57.485526 master-1 kubenswrapper[4771]: W1011 10:53:57.485459 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a75aab2_123f_491f_af07_939ade33aadc.slice/crio-c4bf6c763f65cec074c6888894519b5a728d8d1eb1da7613bf076459d93a73cc WatchSource:0}: Error finding container c4bf6c763f65cec074c6888894519b5a728d8d1eb1da7613bf076459d93a73cc: Status 404 returned error can't find the container with id c4bf6c763f65cec074c6888894519b5a728d8d1eb1da7613bf076459d93a73cc Oct 11 10:53:57.640645 master-1 kubenswrapper[4771]: I1011 10:53:57.640564 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7-config-292gb" event={"ID":"5a75aab2-123f-491f-af07-939ade33aadc","Type":"ContainerStarted","Data":"c4bf6c763f65cec074c6888894519b5a728d8d1eb1da7613bf076459d93a73cc"} Oct 11 10:53:58.654201 master-1 kubenswrapper[4771]: I1011 10:53:58.654107 4771 generic.go:334] "Generic (PLEG): container finished" podID="5a75aab2-123f-491f-af07-939ade33aadc" containerID="b9f9706961ea78a9f4e52f8e9ebb80aedc250ae90f55ef49b6cd39d2d53a0f62" exitCode=0 Oct 11 10:53:58.654201 master-1 kubenswrapper[4771]: I1011 10:53:58.654190 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7-config-292gb" event={"ID":"5a75aab2-123f-491f-af07-939ade33aadc","Type":"ContainerDied","Data":"b9f9706961ea78a9f4e52f8e9ebb80aedc250ae90f55ef49b6cd39d2d53a0f62"} Oct 11 10:54:00.122249 master-1 kubenswrapper[4771]: I1011 10:54:00.122102 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:54:00.241942 master-1 kubenswrapper[4771]: I1011 10:54:00.241845 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-scripts\") pod \"5a75aab2-123f-491f-af07-939ade33aadc\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " Oct 11 10:54:00.242314 master-1 kubenswrapper[4771]: I1011 10:54:00.241983 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run\") pod \"5a75aab2-123f-491f-af07-939ade33aadc\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " Oct 11 10:54:00.242314 master-1 kubenswrapper[4771]: I1011 10:54:00.242033 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run-ovn\") pod \"5a75aab2-123f-491f-af07-939ade33aadc\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " Oct 11 10:54:00.242314 master-1 kubenswrapper[4771]: I1011 10:54:00.242072 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-additional-scripts\") pod \"5a75aab2-123f-491f-af07-939ade33aadc\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " Oct 11 10:54:00.242314 master-1 kubenswrapper[4771]: I1011 10:54:00.242122 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run" (OuterVolumeSpecName: "var-run") pod "5a75aab2-123f-491f-af07-939ade33aadc" (UID: "5a75aab2-123f-491f-af07-939ade33aadc"). InnerVolumeSpecName "var-run". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:54:00.242314 master-1 kubenswrapper[4771]: I1011 10:54:00.242156 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-log-ovn\") pod \"5a75aab2-123f-491f-af07-939ade33aadc\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " Oct 11 10:54:00.242314 master-1 kubenswrapper[4771]: I1011 10:54:00.242257 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg8cm\" (UniqueName: \"kubernetes.io/projected/5a75aab2-123f-491f-af07-939ade33aadc-kube-api-access-jg8cm\") pod \"5a75aab2-123f-491f-af07-939ade33aadc\" (UID: \"5a75aab2-123f-491f-af07-939ade33aadc\") " Oct 11 10:54:00.242973 master-1 kubenswrapper[4771]: I1011 10:54:00.242140 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run-ovn" (OuterVolumeSpecName: "var-run-ovn") pod "5a75aab2-123f-491f-af07-939ade33aadc" (UID: "5a75aab2-123f-491f-af07-939ade33aadc"). InnerVolumeSpecName "var-run-ovn". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:54:00.242973 master-1 kubenswrapper[4771]: I1011 10:54:00.242210 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-log-ovn" (OuterVolumeSpecName: "var-log-ovn") pod "5a75aab2-123f-491f-af07-939ade33aadc" (UID: "5a75aab2-123f-491f-af07-939ade33aadc"). InnerVolumeSpecName "var-log-ovn". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:54:00.242973 master-1 kubenswrapper[4771]: I1011 10:54:00.242932 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-additional-scripts" (OuterVolumeSpecName: "additional-scripts") pod "5a75aab2-123f-491f-af07-939ade33aadc" (UID: "5a75aab2-123f-491f-af07-939ade33aadc"). InnerVolumeSpecName "additional-scripts". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:00.243265 master-1 kubenswrapper[4771]: I1011 10:54:00.243151 4771 reconciler_common.go:293] "Volume detached for volume \"additional-scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-additional-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:00.243265 master-1 kubenswrapper[4771]: I1011 10:54:00.243169 4771 reconciler_common.go:293] "Volume detached for volume \"var-log-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-log-ovn\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:00.243265 master-1 kubenswrapper[4771]: I1011 10:54:00.243179 4771 reconciler_common.go:293] "Volume detached for volume \"var-run\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:00.243265 master-1 kubenswrapper[4771]: I1011 10:54:00.243189 4771 reconciler_common.go:293] "Volume detached for volume \"var-run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a75aab2-123f-491f-af07-939ade33aadc-var-run-ovn\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:00.244259 master-1 kubenswrapper[4771]: I1011 10:54:00.244153 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-scripts" (OuterVolumeSpecName: "scripts") pod "5a75aab2-123f-491f-af07-939ade33aadc" (UID: "5a75aab2-123f-491f-af07-939ade33aadc"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:00.247624 master-1 kubenswrapper[4771]: I1011 10:54:00.247559 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a75aab2-123f-491f-af07-939ade33aadc-kube-api-access-jg8cm" (OuterVolumeSpecName: "kube-api-access-jg8cm") pod "5a75aab2-123f-491f-af07-939ade33aadc" (UID: "5a75aab2-123f-491f-af07-939ade33aadc"). InnerVolumeSpecName "kube-api-access-jg8cm". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:54:00.346033 master-1 kubenswrapper[4771]: I1011 10:54:00.345896 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-jg8cm\" (UniqueName: \"kubernetes.io/projected/5a75aab2-123f-491f-af07-939ade33aadc-kube-api-access-jg8cm\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:00.346033 master-1 kubenswrapper[4771]: I1011 10:54:00.346009 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/5a75aab2-123f-491f-af07-939ade33aadc-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:00.677582 master-1 kubenswrapper[4771]: I1011 10:54:00.677452 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-controller-mtzk7-config-292gb" event={"ID":"5a75aab2-123f-491f-af07-939ade33aadc","Type":"ContainerDied","Data":"c4bf6c763f65cec074c6888894519b5a728d8d1eb1da7613bf076459d93a73cc"} Oct 11 10:54:00.677582 master-1 kubenswrapper[4771]: I1011 10:54:00.677557 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-controller-mtzk7-config-292gb" Oct 11 10:54:00.678095 master-1 kubenswrapper[4771]: I1011 10:54:00.677567 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4bf6c763f65cec074c6888894519b5a728d8d1eb1da7613bf076459d93a73cc" Oct 11 10:54:01.258727 master-1 kubenswrapper[4771]: I1011 10:54:01.258629 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ovn-controller-mtzk7-config-292gb"] Oct 11 10:54:01.269535 master-1 kubenswrapper[4771]: I1011 10:54:01.269459 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ovn-controller-mtzk7-config-292gb"] Oct 11 10:54:02.452569 master-1 kubenswrapper[4771]: I1011 10:54:02.452466 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a75aab2-123f-491f-af07-939ade33aadc" path="/var/lib/kubelet/pods/5a75aab2-123f-491f-af07-939ade33aadc/volumes" Oct 11 10:54:02.899989 master-1 kubenswrapper[4771]: I1011 10:54:02.898962 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:02.903191 master-1 kubenswrapper[4771]: I1011 10:54:02.903103 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:02.950084 master-1 kubenswrapper[4771]: I1011 10:54:02.950016 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:54:03.048184 master-1 kubenswrapper[4771]: I1011 10:54:03.048023 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-7b64bc6b99-wp674"] Oct 11 10:54:03.049211 master-1 kubenswrapper[4771]: I1011 10:54:03.049144 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" podUID="49a589aa-7d75-4aba-aca3-9fffa3d86378" containerName="dnsmasq-dns" containerID="cri-o://e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db" gracePeriod=10 Oct 11 10:54:03.654927 master-1 kubenswrapper[4771]: I1011 10:54:03.654749 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:54:03.712534 master-1 kubenswrapper[4771]: I1011 10:54:03.711473 4771 generic.go:334] "Generic (PLEG): container finished" podID="49a589aa-7d75-4aba-aca3-9fffa3d86378" containerID="e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db" exitCode=0 Oct 11 10:54:03.712534 master-1 kubenswrapper[4771]: I1011 10:54:03.711561 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" Oct 11 10:54:03.712534 master-1 kubenswrapper[4771]: I1011 10:54:03.711608 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" event={"ID":"49a589aa-7d75-4aba-aca3-9fffa3d86378","Type":"ContainerDied","Data":"e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db"} Oct 11 10:54:03.712534 master-1 kubenswrapper[4771]: I1011 10:54:03.711737 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-7b64bc6b99-wp674" event={"ID":"49a589aa-7d75-4aba-aca3-9fffa3d86378","Type":"ContainerDied","Data":"e04ccf4542c4af977ce52340f83783b293bb99776af706005aa7ec0a114852af"} Oct 11 10:54:03.712534 master-1 kubenswrapper[4771]: I1011 10:54:03.711781 4771 scope.go:117] "RemoveContainer" containerID="e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db" Oct 11 10:54:03.714501 master-1 kubenswrapper[4771]: I1011 10:54:03.714464 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:03.745490 master-1 kubenswrapper[4771]: I1011 10:54:03.745428 4771 scope.go:117] "RemoveContainer" containerID="9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f" Oct 11 10:54:03.788182 master-1 kubenswrapper[4771]: I1011 10:54:03.788102 4771 scope.go:117] "RemoveContainer" containerID="e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db" Oct 11 10:54:03.788892 master-1 kubenswrapper[4771]: E1011 10:54:03.788843 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db\": container with ID starting with e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db not found: ID does not exist" containerID="e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db" Oct 11 10:54:03.789061 master-1 kubenswrapper[4771]: I1011 10:54:03.788889 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db"} err="failed to get container status \"e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db\": rpc error: code = NotFound desc = could not find container \"e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db\": container with ID starting with e888654814b1c6bb30b9a08fe67a78675ceee7885666f9f7dfa7244292fa27db not found: ID does not exist" Oct 11 10:54:03.789061 master-1 kubenswrapper[4771]: I1011 10:54:03.788923 4771 scope.go:117] "RemoveContainer" containerID="9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f" Oct 11 10:54:03.789821 master-1 kubenswrapper[4771]: E1011 10:54:03.789733 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f\": container with ID starting with 9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f not found: ID does not exist" containerID="9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f" Oct 11 10:54:03.789971 master-1 kubenswrapper[4771]: I1011 10:54:03.789818 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f"} err="failed to get container status \"9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f\": rpc error: code = NotFound desc = could not find container \"9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f\": container with ID starting with 9c117e659394937d3c04e2bfcad7f394e379e5635acae77fe7dccd35b074376f not found: ID does not exist" Oct 11 10:54:03.835218 master-1 kubenswrapper[4771]: I1011 10:54:03.835148 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-dns-svc\") pod \"49a589aa-7d75-4aba-aca3-9fffa3d86378\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " Oct 11 10:54:03.835218 master-1 kubenswrapper[4771]: I1011 10:54:03.835195 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-config\") pod \"49a589aa-7d75-4aba-aca3-9fffa3d86378\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " Oct 11 10:54:03.835218 master-1 kubenswrapper[4771]: I1011 10:54:03.835235 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgbhb\" (UniqueName: \"kubernetes.io/projected/49a589aa-7d75-4aba-aca3-9fffa3d86378-kube-api-access-lgbhb\") pod \"49a589aa-7d75-4aba-aca3-9fffa3d86378\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " Oct 11 10:54:03.835655 master-1 kubenswrapper[4771]: I1011 10:54:03.835284 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-nb\") pod \"49a589aa-7d75-4aba-aca3-9fffa3d86378\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " Oct 11 10:54:03.835655 master-1 kubenswrapper[4771]: I1011 10:54:03.835370 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-sb\") pod \"49a589aa-7d75-4aba-aca3-9fffa3d86378\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " Oct 11 10:54:03.840427 master-1 kubenswrapper[4771]: I1011 10:54:03.840271 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a589aa-7d75-4aba-aca3-9fffa3d86378-kube-api-access-lgbhb" (OuterVolumeSpecName: "kube-api-access-lgbhb") pod "49a589aa-7d75-4aba-aca3-9fffa3d86378" (UID: "49a589aa-7d75-4aba-aca3-9fffa3d86378"). InnerVolumeSpecName "kube-api-access-lgbhb". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:54:03.869021 master-1 kubenswrapper[4771]: I1011 10:54:03.868906 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-sb" (OuterVolumeSpecName: "ovsdbserver-sb") pod "49a589aa-7d75-4aba-aca3-9fffa3d86378" (UID: "49a589aa-7d75-4aba-aca3-9fffa3d86378"). InnerVolumeSpecName "ovsdbserver-sb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:03.883186 master-1 kubenswrapper[4771]: I1011 10:54:03.883099 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-nb" (OuterVolumeSpecName: "ovsdbserver-nb") pod "49a589aa-7d75-4aba-aca3-9fffa3d86378" (UID: "49a589aa-7d75-4aba-aca3-9fffa3d86378"). InnerVolumeSpecName "ovsdbserver-nb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:03.902275 master-1 kubenswrapper[4771]: E1011 10:54:03.901718 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-config podName:49a589aa-7d75-4aba-aca3-9fffa3d86378 nodeName:}" failed. No retries permitted until 2025-10-11 10:54:04.401683652 +0000 UTC m=+1676.375910113 (durationBeforeRetry 500ms). Error: error cleaning subPath mounts for volume "config" (UniqueName: "kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-config") pod "49a589aa-7d75-4aba-aca3-9fffa3d86378" (UID: "49a589aa-7d75-4aba-aca3-9fffa3d86378") : error deleting /var/lib/kubelet/pods/49a589aa-7d75-4aba-aca3-9fffa3d86378/volume-subpaths: remove /var/lib/kubelet/pods/49a589aa-7d75-4aba-aca3-9fffa3d86378/volume-subpaths: no such file or directory Oct 11 10:54:03.902275 master-1 kubenswrapper[4771]: I1011 10:54:03.902112 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "49a589aa-7d75-4aba-aca3-9fffa3d86378" (UID: "49a589aa-7d75-4aba-aca3-9fffa3d86378"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:03.937609 master-1 kubenswrapper[4771]: I1011 10:54:03.937552 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:03.937609 master-1 kubenswrapper[4771]: I1011 10:54:03.937593 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-lgbhb\" (UniqueName: \"kubernetes.io/projected/49a589aa-7d75-4aba-aca3-9fffa3d86378-kube-api-access-lgbhb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:03.937609 master-1 kubenswrapper[4771]: I1011 10:54:03.937605 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-nb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:03.937609 master-1 kubenswrapper[4771]: I1011 10:54:03.937614 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-ovsdbserver-sb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:04.450236 master-1 kubenswrapper[4771]: I1011 10:54:04.450176 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-config\") pod \"49a589aa-7d75-4aba-aca3-9fffa3d86378\" (UID: \"49a589aa-7d75-4aba-aca3-9fffa3d86378\") " Oct 11 10:54:04.450982 master-1 kubenswrapper[4771]: I1011 10:54:04.450942 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-config" (OuterVolumeSpecName: "config") pod "49a589aa-7d75-4aba-aca3-9fffa3d86378" (UID: "49a589aa-7d75-4aba-aca3-9fffa3d86378"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:04.455437 master-1 kubenswrapper[4771]: I1011 10:54:04.455404 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/rabbitmq-cell1-server-0" Oct 11 10:54:04.552200 master-1 kubenswrapper[4771]: I1011 10:54:04.552109 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49a589aa-7d75-4aba-aca3-9fffa3d86378-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:04.675453 master-1 kubenswrapper[4771]: I1011 10:54:04.675400 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-7b64bc6b99-wp674"] Oct 11 10:54:04.684663 master-1 kubenswrapper[4771]: I1011 10:54:04.684602 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-7b64bc6b99-wp674"] Oct 11 10:54:06.456638 master-1 kubenswrapper[4771]: I1011 10:54:06.456541 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49a589aa-7d75-4aba-aca3-9fffa3d86378" path="/var/lib/kubelet/pods/49a589aa-7d75-4aba-aca3-9fffa3d86378/volumes" Oct 11 10:54:06.973050 master-1 kubenswrapper[4771]: I1011 10:54:06.972917 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/prometheus-metric-storage-0"] Oct 11 10:54:06.973295 master-1 kubenswrapper[4771]: I1011 10:54:06.973212 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/prometheus-metric-storage-0" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="prometheus" containerID="cri-o://dceeb58fe69a42771858429ecb1c63834e43aa17881864b8125d37688c790df5" gracePeriod=600 Oct 11 10:54:06.973405 master-1 kubenswrapper[4771]: I1011 10:54:06.973322 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/prometheus-metric-storage-0" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="thanos-sidecar" containerID="cri-o://92f430c78bc4b24a68001349b3a9ac48c77542314f208344451dc9fc116683d7" gracePeriod=600 Oct 11 10:54:06.973492 master-1 kubenswrapper[4771]: I1011 10:54:06.973322 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/prometheus-metric-storage-0" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="config-reloader" containerID="cri-o://71476de09aca12c350c9bd4fa53bcc64aae8fab0a0998dc69e0421858e6b79c4" gracePeriod=600 Oct 11 10:54:07.009648 master-1 kubenswrapper[4771]: I1011 10:54:07.009590 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/rabbitmq-server-0" Oct 11 10:54:07.769719 master-1 kubenswrapper[4771]: I1011 10:54:07.768863 4771 generic.go:334] "Generic (PLEG): container finished" podID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerID="92f430c78bc4b24a68001349b3a9ac48c77542314f208344451dc9fc116683d7" exitCode=0 Oct 11 10:54:07.769719 master-1 kubenswrapper[4771]: I1011 10:54:07.768911 4771 generic.go:334] "Generic (PLEG): container finished" podID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerID="71476de09aca12c350c9bd4fa53bcc64aae8fab0a0998dc69e0421858e6b79c4" exitCode=0 Oct 11 10:54:07.769719 master-1 kubenswrapper[4771]: I1011 10:54:07.768921 4771 generic.go:334] "Generic (PLEG): container finished" podID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerID="dceeb58fe69a42771858429ecb1c63834e43aa17881864b8125d37688c790df5" exitCode=0 Oct 11 10:54:07.769719 master-1 kubenswrapper[4771]: I1011 10:54:07.768950 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerDied","Data":"92f430c78bc4b24a68001349b3a9ac48c77542314f208344451dc9fc116683d7"} Oct 11 10:54:07.769719 master-1 kubenswrapper[4771]: I1011 10:54:07.768984 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerDied","Data":"71476de09aca12c350c9bd4fa53bcc64aae8fab0a0998dc69e0421858e6b79c4"} Oct 11 10:54:07.769719 master-1 kubenswrapper[4771]: I1011 10:54:07.768998 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerDied","Data":"dceeb58fe69a42771858429ecb1c63834e43aa17881864b8125d37688c790df5"} Oct 11 10:54:08.046279 master-1 kubenswrapper[4771]: I1011 10:54:08.046239 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.131056 master-1 kubenswrapper[4771]: I1011 10:54:08.130991 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-tls-assets\") pod \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " Oct 11 10:54:08.131305 master-1 kubenswrapper[4771]: I1011 10:54:08.131076 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config-out\") pod \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " Oct 11 10:54:08.131365 master-1 kubenswrapper[4771]: I1011 10:54:08.131312 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"prometheus-metric-storage-db\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") pod \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " Oct 11 10:54:08.131497 master-1 kubenswrapper[4771]: I1011 10:54:08.131464 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq956\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-kube-api-access-lq956\") pod \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " Oct 11 10:54:08.131543 master-1 kubenswrapper[4771]: I1011 10:54:08.131507 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config\") pod \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " Oct 11 10:54:08.131613 master-1 kubenswrapper[4771]: I1011 10:54:08.131585 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-web-config\") pod \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " Oct 11 10:54:08.131693 master-1 kubenswrapper[4771]: I1011 10:54:08.131665 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"prometheus-metric-storage-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-prometheus-metric-storage-rulefiles-0\") pod \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " Oct 11 10:54:08.131746 master-1 kubenswrapper[4771]: I1011 10:54:08.131724 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-thanos-prometheus-http-client-file\") pod \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\" (UID: \"11c30d1f-16d5-4106-bfae-e6c2d2f64f13\") " Oct 11 10:54:08.134217 master-1 kubenswrapper[4771]: I1011 10:54:08.134159 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-prometheus-metric-storage-rulefiles-0" (OuterVolumeSpecName: "prometheus-metric-storage-rulefiles-0") pod "11c30d1f-16d5-4106-bfae-e6c2d2f64f13" (UID: "11c30d1f-16d5-4106-bfae-e6c2d2f64f13"). InnerVolumeSpecName "prometheus-metric-storage-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:08.134775 master-1 kubenswrapper[4771]: I1011 10:54:08.134736 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "11c30d1f-16d5-4106-bfae-e6c2d2f64f13" (UID: "11c30d1f-16d5-4106-bfae-e6c2d2f64f13"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:54:08.138842 master-1 kubenswrapper[4771]: I1011 10:54:08.138745 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config-out" (OuterVolumeSpecName: "config-out") pod "11c30d1f-16d5-4106-bfae-e6c2d2f64f13" (UID: "11c30d1f-16d5-4106-bfae-e6c2d2f64f13"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:54:08.139264 master-1 kubenswrapper[4771]: I1011 10:54:08.139202 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "11c30d1f-16d5-4106-bfae-e6c2d2f64f13" (UID: "11c30d1f-16d5-4106-bfae-e6c2d2f64f13"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:08.139472 master-1 kubenswrapper[4771]: I1011 10:54:08.139433 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config" (OuterVolumeSpecName: "config") pod "11c30d1f-16d5-4106-bfae-e6c2d2f64f13" (UID: "11c30d1f-16d5-4106-bfae-e6c2d2f64f13"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:08.156020 master-1 kubenswrapper[4771]: I1011 10:54:08.155989 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-web-config" (OuterVolumeSpecName: "web-config") pod "11c30d1f-16d5-4106-bfae-e6c2d2f64f13" (UID: "11c30d1f-16d5-4106-bfae-e6c2d2f64f13"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:08.157007 master-1 kubenswrapper[4771]: I1011 10:54:08.156938 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-kube-api-access-lq956" (OuterVolumeSpecName: "kube-api-access-lq956") pod "11c30d1f-16d5-4106-bfae-e6c2d2f64f13" (UID: "11c30d1f-16d5-4106-bfae-e6c2d2f64f13"). InnerVolumeSpecName "kube-api-access-lq956". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:54:08.167338 master-1 kubenswrapper[4771]: I1011 10:54:08.167027 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2" (OuterVolumeSpecName: "prometheus-metric-storage-db") pod "11c30d1f-16d5-4106-bfae-e6c2d2f64f13" (UID: "11c30d1f-16d5-4106-bfae-e6c2d2f64f13"). InnerVolumeSpecName "pvc-d985c5d6-9363-4eef-9640-f61388292365". PluginName "kubernetes.io/csi", VolumeGidValue "" Oct 11 10:54:08.234465 master-1 kubenswrapper[4771]: I1011 10:54:08.234036 4771 reconciler_common.go:293] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-web-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:08.234465 master-1 kubenswrapper[4771]: I1011 10:54:08.234076 4771 reconciler_common.go:293] "Volume detached for volume \"prometheus-metric-storage-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-prometheus-metric-storage-rulefiles-0\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:08.234465 master-1 kubenswrapper[4771]: I1011 10:54:08.234090 4771 reconciler_common.go:293] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-thanos-prometheus-http-client-file\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:08.234465 master-1 kubenswrapper[4771]: I1011 10:54:08.234103 4771 reconciler_common.go:293] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-tls-assets\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:08.234465 master-1 kubenswrapper[4771]: I1011 10:54:08.234114 4771 reconciler_common.go:293] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config-out\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:08.234465 master-1 kubenswrapper[4771]: I1011 10:54:08.234155 4771 reconciler_common.go:286] "operationExecutor.UnmountDevice started for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") on node \"master-1\" " Oct 11 10:54:08.234465 master-1 kubenswrapper[4771]: I1011 10:54:08.234167 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-lq956\" (UniqueName: \"kubernetes.io/projected/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-kube-api-access-lq956\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:08.234465 master-1 kubenswrapper[4771]: I1011 10:54:08.234176 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/11c30d1f-16d5-4106-bfae-e6c2d2f64f13-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:08.248292 master-1 kubenswrapper[4771]: I1011 10:54:08.248248 4771 csi_attacher.go:630] kubernetes.io/csi: attacher.UnmountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping UnmountDevice... Oct 11 10:54:08.248530 master-1 kubenswrapper[4771]: I1011 10:54:08.248428 4771 operation_generator.go:917] UnmountDevice succeeded for volume "pvc-d985c5d6-9363-4eef-9640-f61388292365" (UniqueName: "kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2") on node "master-1" Oct 11 10:54:08.336866 master-1 kubenswrapper[4771]: I1011 10:54:08.336753 4771 reconciler_common.go:293] "Volume detached for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:08.777683 master-1 kubenswrapper[4771]: I1011 10:54:08.777547 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"11c30d1f-16d5-4106-bfae-e6c2d2f64f13","Type":"ContainerDied","Data":"b089e8e4b13a2f28ac29e4b38aaf2c96910827eb74888a884dedec830020d9fe"} Oct 11 10:54:08.777683 master-1 kubenswrapper[4771]: I1011 10:54:08.777617 4771 scope.go:117] "RemoveContainer" containerID="92f430c78bc4b24a68001349b3a9ac48c77542314f208344451dc9fc116683d7" Oct 11 10:54:08.778617 master-1 kubenswrapper[4771]: I1011 10:54:08.778582 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.799890 master-1 kubenswrapper[4771]: I1011 10:54:08.799820 4771 scope.go:117] "RemoveContainer" containerID="71476de09aca12c350c9bd4fa53bcc64aae8fab0a0998dc69e0421858e6b79c4" Oct 11 10:54:08.813952 master-1 kubenswrapper[4771]: I1011 10:54:08.812728 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/prometheus-metric-storage-0"] Oct 11 10:54:08.819445 master-1 kubenswrapper[4771]: I1011 10:54:08.819412 4771 scope.go:117] "RemoveContainer" containerID="dceeb58fe69a42771858429ecb1c63834e43aa17881864b8125d37688c790df5" Oct 11 10:54:08.822136 master-1 kubenswrapper[4771]: I1011 10:54:08.822107 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/prometheus-metric-storage-0"] Oct 11 10:54:08.850618 master-1 kubenswrapper[4771]: I1011 10:54:08.850461 4771 scope.go:117] "RemoveContainer" containerID="1023c854292d2503211da52aaf16aa7e2199948c97ebed99bad537459ca3e33b" Oct 11 10:54:08.853684 master-1 kubenswrapper[4771]: I1011 10:54:08.853642 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/prometheus-metric-storage-0"] Oct 11 10:54:08.853989 master-1 kubenswrapper[4771]: E1011 10:54:08.853962 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="49a589aa-7d75-4aba-aca3-9fffa3d86378" containerName="dnsmasq-dns" Oct 11 10:54:08.853989 master-1 kubenswrapper[4771]: I1011 10:54:08.853986 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a589aa-7d75-4aba-aca3-9fffa3d86378" containerName="dnsmasq-dns" Oct 11 10:54:08.853989 master-1 kubenswrapper[4771]: E1011 10:54:08.854002 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5a75aab2-123f-491f-af07-939ade33aadc" containerName="ovn-config" Oct 11 10:54:08.853989 master-1 kubenswrapper[4771]: I1011 10:54:08.854012 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a75aab2-123f-491f-af07-939ade33aadc" containerName="ovn-config" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: E1011 10:54:08.854026 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="49a589aa-7d75-4aba-aca3-9fffa3d86378" containerName="init" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: I1011 10:54:08.854037 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a589aa-7d75-4aba-aca3-9fffa3d86378" containerName="init" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: E1011 10:54:08.854046 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="thanos-sidecar" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: I1011 10:54:08.854054 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="thanos-sidecar" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: E1011 10:54:08.854080 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="init-config-reloader" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: I1011 10:54:08.854089 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="init-config-reloader" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: E1011 10:54:08.854108 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="prometheus" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: I1011 10:54:08.854116 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="prometheus" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: E1011 10:54:08.854127 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="config-reloader" Oct 11 10:54:08.854238 master-1 kubenswrapper[4771]: I1011 10:54:08.854137 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="config-reloader" Oct 11 10:54:08.855028 master-1 kubenswrapper[4771]: I1011 10:54:08.854278 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="thanos-sidecar" Oct 11 10:54:08.855028 master-1 kubenswrapper[4771]: I1011 10:54:08.854294 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="49a589aa-7d75-4aba-aca3-9fffa3d86378" containerName="dnsmasq-dns" Oct 11 10:54:08.855028 master-1 kubenswrapper[4771]: I1011 10:54:08.854308 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5a75aab2-123f-491f-af07-939ade33aadc" containerName="ovn-config" Oct 11 10:54:08.855028 master-1 kubenswrapper[4771]: I1011 10:54:08.854320 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="prometheus" Oct 11 10:54:08.855028 master-1 kubenswrapper[4771]: I1011 10:54:08.854330 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="config-reloader" Oct 11 10:54:08.856001 master-1 kubenswrapper[4771]: I1011 10:54:08.855972 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.861913 master-1 kubenswrapper[4771]: I1011 10:54:08.861400 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"prometheus-metric-storage-rulefiles-0" Oct 11 10:54:08.861913 master-1 kubenswrapper[4771]: I1011 10:54:08.861647 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"prometheus-metric-storage-web-config" Oct 11 10:54:08.862124 master-1 kubenswrapper[4771]: I1011 10:54:08.861699 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-metric-storage-prometheus-svc" Oct 11 10:54:08.865520 master-1 kubenswrapper[4771]: I1011 10:54:08.863365 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"prometheus-metric-storage" Oct 11 10:54:08.865520 master-1 kubenswrapper[4771]: I1011 10:54:08.863890 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"prometheus-metric-storage-thanos-prometheus-http-client-file" Oct 11 10:54:08.876447 master-1 kubenswrapper[4771]: I1011 10:54:08.876261 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"prometheus-metric-storage-tls-assets-0" Oct 11 10:54:08.883309 master-1 kubenswrapper[4771]: I1011 10:54:08.883130 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/prometheus-metric-storage-0"] Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.948853 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-config\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.948912 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmc9\" (UniqueName: \"kubernetes.io/projected/abb599b9-db44-492a-bc73-6ff5a2c212d5-kube-api-access-jnmc9\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.948944 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-cert-metric-storage-promethe-dc638c2d\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-web-config-tls-secret-key-cert-metric-storage-promethe-dc638c2d\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.948962 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-web-config\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.948987 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-thanos-prometheus-http-client-file\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.949013 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-cert-metric-storage-prometh-dc638c2d\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-web-config-tls-secret-cert-cert-metric-storage-prometh-dc638c2d\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.949036 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/abb599b9-db44-492a-bc73-6ff5a2c212d5-tls-assets\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.949060 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-metric-storage-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/abb599b9-db44-492a-bc73-6ff5a2c212d5-prometheus-metric-storage-rulefiles-0\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.949092 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/abb599b9-db44-492a-bc73-6ff5a2c212d5-config-out\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.949115 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:08.951380 master-1 kubenswrapper[4771]: I1011 10:54:08.949143 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-secret-combined-ca-bundle\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062550 master-1 kubenswrapper[4771]: I1011 10:54:09.062441 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-config\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062728 master-1 kubenswrapper[4771]: I1011 10:54:09.062580 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmc9\" (UniqueName: \"kubernetes.io/projected/abb599b9-db44-492a-bc73-6ff5a2c212d5-kube-api-access-jnmc9\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062728 master-1 kubenswrapper[4771]: I1011 10:54:09.062630 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-cert-metric-storage-promethe-dc638c2d\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-web-config-tls-secret-key-cert-metric-storage-promethe-dc638c2d\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062728 master-1 kubenswrapper[4771]: I1011 10:54:09.062664 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-web-config\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062728 master-1 kubenswrapper[4771]: I1011 10:54:09.062691 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-thanos-prometheus-http-client-file\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062867 master-1 kubenswrapper[4771]: I1011 10:54:09.062733 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-cert-metric-storage-prometh-dc638c2d\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-web-config-tls-secret-cert-cert-metric-storage-prometh-dc638c2d\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062867 master-1 kubenswrapper[4771]: I1011 10:54:09.062763 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/abb599b9-db44-492a-bc73-6ff5a2c212d5-tls-assets\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062867 master-1 kubenswrapper[4771]: I1011 10:54:09.062791 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"prometheus-metric-storage-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/abb599b9-db44-492a-bc73-6ff5a2c212d5-prometheus-metric-storage-rulefiles-0\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062867 master-1 kubenswrapper[4771]: I1011 10:54:09.062828 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/abb599b9-db44-492a-bc73-6ff5a2c212d5-config-out\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.062867 master-1 kubenswrapper[4771]: I1011 10:54:09.062857 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.063016 master-1 kubenswrapper[4771]: I1011 10:54:09.062886 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-secret-combined-ca-bundle\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.068123 master-1 kubenswrapper[4771]: I1011 10:54:09.068086 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-cert-metric-storage-promethe-dc638c2d\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-web-config-tls-secret-key-cert-metric-storage-promethe-dc638c2d\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.068123 master-1 kubenswrapper[4771]: I1011 10:54:09.068122 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-secret-combined-ca-bundle\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.068406 master-1 kubenswrapper[4771]: I1011 10:54:09.068371 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-config\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.069171 master-1 kubenswrapper[4771]: I1011 10:54:09.068803 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"prometheus-metric-storage-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/abb599b9-db44-492a-bc73-6ff5a2c212d5-prometheus-metric-storage-rulefiles-0\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.069535 master-1 kubenswrapper[4771]: I1011 10:54:09.069504 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/abb599b9-db44-492a-bc73-6ff5a2c212d5-tls-assets\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.073016 master-1 kubenswrapper[4771]: I1011 10:54:09.072053 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/abb599b9-db44-492a-bc73-6ff5a2c212d5-config-out\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.073016 master-1 kubenswrapper[4771]: I1011 10:54:09.072965 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-cert-metric-storage-prometh-dc638c2d\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-web-config-tls-secret-cert-cert-metric-storage-prometh-dc638c2d\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.073682 master-1 kubenswrapper[4771]: I1011 10:54:09.073629 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-web-config\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.073886 master-1 kubenswrapper[4771]: I1011 10:54:09.073843 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/abb599b9-db44-492a-bc73-6ff5a2c212d5-thanos-prometheus-http-client-file\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.074382 master-1 kubenswrapper[4771]: I1011 10:54:09.074288 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:54:09.074382 master-1 kubenswrapper[4771]: I1011 10:54:09.074342 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/37699e2b75858a97c8af891d5e1a76727de9abb22a62dc041bfd38b0b8d8c160/globalmount\"" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:09.102342 master-1 kubenswrapper[4771]: I1011 10:54:09.102247 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmc9\" (UniqueName: \"kubernetes.io/projected/abb599b9-db44-492a-bc73-6ff5a2c212d5-kube-api-access-jnmc9\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:10.451725 master-1 kubenswrapper[4771]: I1011 10:54:10.451533 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" path="/var/lib/kubelet/pods/11c30d1f-16d5-4106-bfae-e6c2d2f64f13/volumes" Oct 11 10:54:10.585879 master-1 kubenswrapper[4771]: I1011 10:54:10.585751 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-d985c5d6-9363-4eef-9640-f61388292365\" (UniqueName: \"kubernetes.io/csi/topolvm.io^0990b59f-0f58-467f-8736-1b158b7725d2\") pod \"prometheus-metric-storage-0\" (UID: \"abb599b9-db44-492a-bc73-6ff5a2c212d5\") " pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:10.703571 master-1 kubenswrapper[4771]: I1011 10:54:10.703446 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:10.899839 master-1 kubenswrapper[4771]: I1011 10:54:10.899050 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/prometheus-metric-storage-0" podUID="11c30d1f-16d5-4106-bfae-e6c2d2f64f13" containerName="prometheus" probeResult="failure" output="Get \"http://10.129.0.112:9090/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 11 10:54:11.445987 master-1 kubenswrapper[4771]: I1011 10:54:11.441529 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/prometheus-metric-storage-0"] Oct 11 10:54:11.809122 master-1 kubenswrapper[4771]: I1011 10:54:11.809076 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"abb599b9-db44-492a-bc73-6ff5a2c212d5","Type":"ContainerStarted","Data":"95d32491bd30dbc01dd3b8c79622bf69717fc1d9bfc98c189bf88a06df6c01e7"} Oct 11 10:54:14.836604 master-1 kubenswrapper[4771]: I1011 10:54:14.836519 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"abb599b9-db44-492a-bc73-6ff5a2c212d5","Type":"ContainerStarted","Data":"5e072db00dc200fff13cfbd416cac5a518950f3f680e875e683c4e6353c36aa5"} Oct 11 10:54:22.472714 master-1 kubenswrapper[4771]: I1011 10:54:22.472345 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:54:22.474935 master-1 kubenswrapper[4771]: I1011 10:54:22.474895 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:54:22.477792 master-1 kubenswrapper[4771]: I1011 10:54:22.477404 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 10:54:22.477792 master-1 kubenswrapper[4771]: I1011 10:54:22.477693 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 10:54:22.504343 master-1 kubenswrapper[4771]: I1011 10:54:22.500383 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:54:22.538823 master-1 kubenswrapper[4771]: I1011 10:54:22.538673 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-run-httpd\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.538823 master-1 kubenswrapper[4771]: I1011 10:54:22.538744 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-config-data\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.538823 master-1 kubenswrapper[4771]: I1011 10:54:22.538786 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-log-httpd\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.539147 master-1 kubenswrapper[4771]: I1011 10:54:22.538859 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.539147 master-1 kubenswrapper[4771]: I1011 10:54:22.538887 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sj2\" (UniqueName: \"kubernetes.io/projected/6508f17e-afc7-44dd-89b4-2efa8a124b12-kube-api-access-76sj2\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.539147 master-1 kubenswrapper[4771]: I1011 10:54:22.539121 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-scripts\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.539147 master-1 kubenswrapper[4771]: I1011 10:54:22.539144 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.606156 master-1 kubenswrapper[4771]: I1011 10:54:22.606093 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-595686b98f-blmgp"] Oct 11 10:54:22.607714 master-1 kubenswrapper[4771]: I1011 10:54:22.607608 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.612544 master-1 kubenswrapper[4771]: I1011 10:54:22.612348 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"dns-swift-storage-0" Oct 11 10:54:22.623146 master-1 kubenswrapper[4771]: I1011 10:54:22.623072 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-595686b98f-blmgp"] Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.646767 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.646830 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-76sj2\" (UniqueName: \"kubernetes.io/projected/6508f17e-afc7-44dd-89b4-2efa8a124b12-kube-api-access-76sj2\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.646910 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-nb\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.646950 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-scripts\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.646977 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.647010 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-config\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.647061 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-swift-storage-0\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.647089 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-run-httpd\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.647117 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-svc\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.647141 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-config-data\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.647177 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-log-httpd\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.647210 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcx9\" (UniqueName: \"kubernetes.io/projected/a50b2fec-a3b6-4245-9080-5987b411b581-kube-api-access-mtcx9\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.647490 master-1 kubenswrapper[4771]: I1011 10:54:22.647239 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-sb\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.652783 master-1 kubenswrapper[4771]: I1011 10:54:22.652695 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.653223 master-1 kubenswrapper[4771]: I1011 10:54:22.653015 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-log-httpd\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.653387 master-1 kubenswrapper[4771]: I1011 10:54:22.653328 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-scripts\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.654547 master-1 kubenswrapper[4771]: I1011 10:54:22.654334 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-run-httpd\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.655730 master-1 kubenswrapper[4771]: I1011 10:54:22.655661 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.655819 master-1 kubenswrapper[4771]: I1011 10:54:22.655721 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-config-data\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.675076 master-1 kubenswrapper[4771]: I1011 10:54:22.674983 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sj2\" (UniqueName: \"kubernetes.io/projected/6508f17e-afc7-44dd-89b4-2efa8a124b12-kube-api-access-76sj2\") pod \"ceilometer-0\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " pod="openstack/ceilometer-0" Oct 11 10:54:22.749225 master-1 kubenswrapper[4771]: I1011 10:54:22.749160 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-nb\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.749570 master-1 kubenswrapper[4771]: I1011 10:54:22.749252 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-config\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.749570 master-1 kubenswrapper[4771]: I1011 10:54:22.749295 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-swift-storage-0\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.749570 master-1 kubenswrapper[4771]: I1011 10:54:22.749322 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-svc\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.749570 master-1 kubenswrapper[4771]: I1011 10:54:22.749372 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcx9\" (UniqueName: \"kubernetes.io/projected/a50b2fec-a3b6-4245-9080-5987b411b581-kube-api-access-mtcx9\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.749570 master-1 kubenswrapper[4771]: I1011 10:54:22.749393 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-sb\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.750741 master-1 kubenswrapper[4771]: I1011 10:54:22.750328 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-sb\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.750741 master-1 kubenswrapper[4771]: I1011 10:54:22.750520 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-config\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.750741 master-1 kubenswrapper[4771]: I1011 10:54:22.750683 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-svc\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.752332 master-1 kubenswrapper[4771]: I1011 10:54:22.752292 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-swift-storage-0\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.753131 master-1 kubenswrapper[4771]: I1011 10:54:22.753094 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-nb\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.772820 master-1 kubenswrapper[4771]: I1011 10:54:22.772750 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcx9\" (UniqueName: \"kubernetes.io/projected/a50b2fec-a3b6-4245-9080-5987b411b581-kube-api-access-mtcx9\") pod \"dnsmasq-dns-595686b98f-blmgp\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:22.789374 master-1 kubenswrapper[4771]: I1011 10:54:22.789315 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:54:22.930464 master-1 kubenswrapper[4771]: I1011 10:54:22.930410 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:23.271416 master-1 kubenswrapper[4771]: I1011 10:54:23.269557 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:54:23.279126 master-1 kubenswrapper[4771]: W1011 10:54:23.279075 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6508f17e_afc7_44dd_89b4_2efa8a124b12.slice/crio-7c53a7f6f217d02a6155b61317bdcecef01a919dcbe718b5d7a4a4096ceec2ae WatchSource:0}: Error finding container 7c53a7f6f217d02a6155b61317bdcecef01a919dcbe718b5d7a4a4096ceec2ae: Status 404 returned error can't find the container with id 7c53a7f6f217d02a6155b61317bdcecef01a919dcbe718b5d7a4a4096ceec2ae Oct 11 10:54:23.366035 master-1 kubenswrapper[4771]: I1011 10:54:23.365928 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-595686b98f-blmgp"] Oct 11 10:54:23.373775 master-1 kubenswrapper[4771]: W1011 10:54:23.373540 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda50b2fec_a3b6_4245_9080_5987b411b581.slice/crio-df3edeb105ed637b9d7fa0933dc5cae9f70ee8feff1cdbfb3585b9bc6889a72c WatchSource:0}: Error finding container df3edeb105ed637b9d7fa0933dc5cae9f70ee8feff1cdbfb3585b9bc6889a72c: Status 404 returned error can't find the container with id df3edeb105ed637b9d7fa0933dc5cae9f70ee8feff1cdbfb3585b9bc6889a72c Oct 11 10:54:23.916855 master-1 kubenswrapper[4771]: I1011 10:54:23.916778 4771 generic.go:334] "Generic (PLEG): container finished" podID="abb599b9-db44-492a-bc73-6ff5a2c212d5" containerID="5e072db00dc200fff13cfbd416cac5a518950f3f680e875e683c4e6353c36aa5" exitCode=0 Oct 11 10:54:23.917606 master-1 kubenswrapper[4771]: I1011 10:54:23.916902 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"abb599b9-db44-492a-bc73-6ff5a2c212d5","Type":"ContainerDied","Data":"5e072db00dc200fff13cfbd416cac5a518950f3f680e875e683c4e6353c36aa5"} Oct 11 10:54:23.918315 master-1 kubenswrapper[4771]: I1011 10:54:23.918264 4771 generic.go:334] "Generic (PLEG): container finished" podID="a50b2fec-a3b6-4245-9080-5987b411b581" containerID="314a76b2857d795a4f3ebe7e8b09e8abca5d105e5ba862e3833d60a9a90b7cc3" exitCode=0 Oct 11 10:54:23.918430 master-1 kubenswrapper[4771]: I1011 10:54:23.918313 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-595686b98f-blmgp" event={"ID":"a50b2fec-a3b6-4245-9080-5987b411b581","Type":"ContainerDied","Data":"314a76b2857d795a4f3ebe7e8b09e8abca5d105e5ba862e3833d60a9a90b7cc3"} Oct 11 10:54:23.918430 master-1 kubenswrapper[4771]: I1011 10:54:23.918381 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-595686b98f-blmgp" event={"ID":"a50b2fec-a3b6-4245-9080-5987b411b581","Type":"ContainerStarted","Data":"df3edeb105ed637b9d7fa0933dc5cae9f70ee8feff1cdbfb3585b9bc6889a72c"} Oct 11 10:54:23.924112 master-1 kubenswrapper[4771]: I1011 10:54:23.923761 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerStarted","Data":"7c53a7f6f217d02a6155b61317bdcecef01a919dcbe718b5d7a4a4096ceec2ae"} Oct 11 10:54:24.933718 master-1 kubenswrapper[4771]: I1011 10:54:24.933653 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"abb599b9-db44-492a-bc73-6ff5a2c212d5","Type":"ContainerStarted","Data":"31dcfe7ad738be71adcd29d247b4f18163e8ed27a14b555fe19eb6dfd4c64dfb"} Oct 11 10:54:24.937444 master-1 kubenswrapper[4771]: I1011 10:54:24.937393 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-595686b98f-blmgp" event={"ID":"a50b2fec-a3b6-4245-9080-5987b411b581","Type":"ContainerStarted","Data":"64768fb3aaa57fbf977b42bcf01d911517cd3d56cc20742d472651a90c1c3f06"} Oct 11 10:54:24.937641 master-1 kubenswrapper[4771]: I1011 10:54:24.937602 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:24.962474 master-1 kubenswrapper[4771]: I1011 10:54:24.962349 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-595686b98f-blmgp" podStartSLOduration=2.962325671 podStartE2EDuration="2.962325671s" podCreationTimestamp="2025-10-11 10:54:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:54:24.961325092 +0000 UTC m=+1696.935551603" watchObservedRunningTime="2025-10-11 10:54:24.962325671 +0000 UTC m=+1696.936552172" Oct 11 10:54:25.317999 master-1 kubenswrapper[4771]: I1011 10:54:25.317914 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:54:25.320290 master-1 kubenswrapper[4771]: I1011 10:54:25.320246 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.325417 master-1 kubenswrapper[4771]: I1011 10:54:25.325368 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-glance-default-public-svc" Oct 11 10:54:25.325643 master-1 kubenswrapper[4771]: I1011 10:54:25.325607 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"glance-scripts" Oct 11 10:54:25.327118 master-1 kubenswrapper[4771]: I1011 10:54:25.327066 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"glance-b5802-default-external-config-data" Oct 11 10:54:25.339997 master-1 kubenswrapper[4771]: I1011 10:54:25.339922 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:54:25.507631 master-1 kubenswrapper[4771]: I1011 10:54:25.507524 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-scripts\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.507631 master-1 kubenswrapper[4771]: I1011 10:54:25.507571 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jlzm\" (UniqueName: \"kubernetes.io/projected/18861a21-406e-479b-8712-9a62ca2ebf4a-kube-api-access-5jlzm\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.507855 master-1 kubenswrapper[4771]: I1011 10:54:25.507662 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-config-data\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.507855 master-1 kubenswrapper[4771]: I1011 10:54:25.507694 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-httpd-run\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.507855 master-1 kubenswrapper[4771]: I1011 10:54:25.507718 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.507855 master-1 kubenswrapper[4771]: I1011 10:54:25.507734 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-logs\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.507855 master-1 kubenswrapper[4771]: I1011 10:54:25.507769 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-public-tls-certs\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.507855 master-1 kubenswrapper[4771]: I1011 10:54:25.507790 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-combined-ca-bundle\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.609444 master-1 kubenswrapper[4771]: I1011 10:54:25.609374 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-scripts\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.609444 master-1 kubenswrapper[4771]: I1011 10:54:25.609430 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5jlzm\" (UniqueName: \"kubernetes.io/projected/18861a21-406e-479b-8712-9a62ca2ebf4a-kube-api-access-5jlzm\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.609772 master-1 kubenswrapper[4771]: I1011 10:54:25.609485 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-config-data\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.609772 master-1 kubenswrapper[4771]: I1011 10:54:25.609512 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-httpd-run\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.609772 master-1 kubenswrapper[4771]: I1011 10:54:25.609533 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.609772 master-1 kubenswrapper[4771]: I1011 10:54:25.609553 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-logs\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.609772 master-1 kubenswrapper[4771]: I1011 10:54:25.609594 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-public-tls-certs\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.609772 master-1 kubenswrapper[4771]: I1011 10:54:25.609615 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-combined-ca-bundle\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.610134 master-1 kubenswrapper[4771]: I1011 10:54:25.610101 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-httpd-run\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.610521 master-1 kubenswrapper[4771]: I1011 10:54:25.610475 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-logs\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.614685 master-1 kubenswrapper[4771]: I1011 10:54:25.614631 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:54:25.614798 master-1 kubenswrapper[4771]: I1011 10:54:25.614696 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/643ba808821ea6db76a2042d255ba68bbc43444ed3cc7e332598424f5540da0c/globalmount\"" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.615618 master-1 kubenswrapper[4771]: I1011 10:54:25.615570 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-combined-ca-bundle\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.616860 master-1 kubenswrapper[4771]: I1011 10:54:25.616823 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-config-data\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.618410 master-1 kubenswrapper[4771]: I1011 10:54:25.618346 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-scripts\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.638322 master-1 kubenswrapper[4771]: I1011 10:54:25.638242 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jlzm\" (UniqueName: \"kubernetes.io/projected/18861a21-406e-479b-8712-9a62ca2ebf4a-kube-api-access-5jlzm\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.717511 master-1 kubenswrapper[4771]: I1011 10:54:25.717400 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-public-tls-certs\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:25.830093 master-1 kubenswrapper[4771]: I1011 10:54:25.830031 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:54:26.327729 master-1 kubenswrapper[4771]: I1011 10:54:26.327669 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:54:26.329898 master-1 kubenswrapper[4771]: I1011 10:54:26.329850 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.332313 master-1 kubenswrapper[4771]: I1011 10:54:26.332271 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"glance-b5802-default-internal-config-data" Oct 11 10:54:26.333662 master-1 kubenswrapper[4771]: I1011 10:54:26.333633 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-glance-default-internal-svc" Oct 11 10:54:26.356565 master-1 kubenswrapper[4771]: I1011 10:54:26.356508 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:54:26.435384 master-1 kubenswrapper[4771]: I1011 10:54:26.430201 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hg2\" (UniqueName: \"kubernetes.io/projected/499f9e94-a738-484d-ae4b-0cc221750d1c-kube-api-access-k5hg2\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.435384 master-1 kubenswrapper[4771]: I1011 10:54:26.430296 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-logs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.435384 master-1 kubenswrapper[4771]: I1011 10:54:26.430335 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-config-data\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.435384 master-1 kubenswrapper[4771]: I1011 10:54:26.430374 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-combined-ca-bundle\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.435384 master-1 kubenswrapper[4771]: I1011 10:54:26.430438 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.435384 master-1 kubenswrapper[4771]: I1011 10:54:26.430490 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-httpd-run\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.435384 master-1 kubenswrapper[4771]: I1011 10:54:26.430567 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-scripts\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.435384 master-1 kubenswrapper[4771]: I1011 10:54:26.430585 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-internal-tls-certs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.536219 master-1 kubenswrapper[4771]: I1011 10:54:26.536127 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-config-data\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.536219 master-1 kubenswrapper[4771]: I1011 10:54:26.536230 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-combined-ca-bundle\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.536523 master-1 kubenswrapper[4771]: I1011 10:54:26.536306 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.536523 master-1 kubenswrapper[4771]: I1011 10:54:26.536443 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-httpd-run\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.536619 master-1 kubenswrapper[4771]: I1011 10:54:26.536529 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-scripts\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.536619 master-1 kubenswrapper[4771]: I1011 10:54:26.536558 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-internal-tls-certs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.536619 master-1 kubenswrapper[4771]: I1011 10:54:26.536595 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hg2\" (UniqueName: \"kubernetes.io/projected/499f9e94-a738-484d-ae4b-0cc221750d1c-kube-api-access-k5hg2\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.536756 master-1 kubenswrapper[4771]: I1011 10:54:26.536692 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-logs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.537525 master-1 kubenswrapper[4771]: I1011 10:54:26.537495 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-logs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.540066 master-1 kubenswrapper[4771]: I1011 10:54:26.539964 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-httpd-run\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.542820 master-1 kubenswrapper[4771]: I1011 10:54:26.542746 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:54:26.542820 master-1 kubenswrapper[4771]: I1011 10:54:26.542798 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/319ddbbf14dc29e9dbd7eec9a997b70a9a11c6eca7f6496495d34ea4ac3ccad0/globalmount\"" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.545500 master-1 kubenswrapper[4771]: I1011 10:54:26.545412 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-config-data\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.547788 master-1 kubenswrapper[4771]: I1011 10:54:26.547746 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-internal-tls-certs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.552939 master-1 kubenswrapper[4771]: I1011 10:54:26.552890 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-combined-ca-bundle\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.560070 master-1 kubenswrapper[4771]: I1011 10:54:26.560010 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hg2\" (UniqueName: \"kubernetes.io/projected/499f9e94-a738-484d-ae4b-0cc221750d1c-kube-api-access-k5hg2\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.564262 master-1 kubenswrapper[4771]: I1011 10:54:26.564188 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-scripts\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:26.977704 master-1 kubenswrapper[4771]: I1011 10:54:26.977618 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"abb599b9-db44-492a-bc73-6ff5a2c212d5","Type":"ContainerStarted","Data":"527c6d5702318963faeb1a8a28a7b7506d42eaf7dcf4a74cf375d67c406096ba"} Oct 11 10:54:27.079935 master-1 kubenswrapper[4771]: I1011 10:54:27.079440 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:27.192630 master-1 kubenswrapper[4771]: I1011 10:54:27.192547 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:28.007301 master-1 kubenswrapper[4771]: I1011 10:54:28.007223 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerStarted","Data":"5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027"} Oct 11 10:54:28.013368 master-1 kubenswrapper[4771]: I1011 10:54:28.011504 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/prometheus-metric-storage-0" event={"ID":"abb599b9-db44-492a-bc73-6ff5a2c212d5","Type":"ContainerStarted","Data":"7285e766541fce2470ed4423abb2e429bac288761eb9a91edaf6349ae518f587"} Oct 11 10:54:28.061605 master-1 kubenswrapper[4771]: I1011 10:54:28.061153 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/prometheus-metric-storage-0" podStartSLOduration=20.061082293 podStartE2EDuration="20.061082293s" podCreationTimestamp="2025-10-11 10:54:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:54:28.055579644 +0000 UTC m=+1700.029806095" watchObservedRunningTime="2025-10-11 10:54:28.061082293 +0000 UTC m=+1700.035308734" Oct 11 10:54:28.170125 master-1 kubenswrapper[4771]: I1011 10:54:28.170069 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:54:28.174832 master-1 kubenswrapper[4771]: W1011 10:54:28.174745 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18861a21_406e_479b_8712_9a62ca2ebf4a.slice/crio-51aa777863a3d17bf81dc45f1659ccef0c9c30b6b9bf5305b555b52a6a626104 WatchSource:0}: Error finding container 51aa777863a3d17bf81dc45f1659ccef0c9c30b6b9bf5305b555b52a6a626104: Status 404 returned error can't find the container with id 51aa777863a3d17bf81dc45f1659ccef0c9c30b6b9bf5305b555b52a6a626104 Oct 11 10:54:28.464452 master-1 kubenswrapper[4771]: I1011 10:54:28.464387 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:28.766406 master-1 kubenswrapper[4771]: I1011 10:54:28.766223 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:29.023615 master-1 kubenswrapper[4771]: I1011 10:54:29.023369 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"18861a21-406e-479b-8712-9a62ca2ebf4a","Type":"ContainerStarted","Data":"51aa777863a3d17bf81dc45f1659ccef0c9c30b6b9bf5305b555b52a6a626104"} Oct 11 10:54:29.632444 master-1 kubenswrapper[4771]: I1011 10:54:29.622437 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:54:30.037849 master-1 kubenswrapper[4771]: I1011 10:54:30.037738 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"499f9e94-a738-484d-ae4b-0cc221750d1c","Type":"ContainerStarted","Data":"ee8592262e70401d099ff1b266023cdb236d7c7195e76597576b3cf0944d23f5"} Oct 11 10:54:30.047677 master-1 kubenswrapper[4771]: I1011 10:54:30.047316 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerStarted","Data":"9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb"} Oct 11 10:54:30.705904 master-1 kubenswrapper[4771]: I1011 10:54:30.704315 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:32.932878 master-1 kubenswrapper[4771]: I1011 10:54:32.932805 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:54:33.010134 master-1 kubenswrapper[4771]: I1011 10:54:33.010072 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-6c99f4877f-vjhdt"] Oct 11 10:54:33.010856 master-1 kubenswrapper[4771]: I1011 10:54:33.010372 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" podUID="30700706-219b-47c1-83cd-278584a3f182" containerName="dnsmasq-dns" containerID="cri-o://2dcdb27cf0dbce506998b4c8cfe73f7847cd892689b076fcba313e976b8a5349" gracePeriod=10 Oct 11 10:54:33.598515 master-1 kubenswrapper[4771]: I1011 10:54:33.598318 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/placement-6b597cbbf8-8j29d"] Oct 11 10:54:33.600169 master-1 kubenswrapper[4771]: I1011 10:54:33.600140 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.603565 master-1 kubenswrapper[4771]: I1011 10:54:33.603521 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"placement-config-data" Oct 11 10:54:33.603679 master-1 kubenswrapper[4771]: I1011 10:54:33.603582 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-placement-internal-svc" Oct 11 10:54:33.603679 master-1 kubenswrapper[4771]: I1011 10:54:33.603578 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-placement-public-svc" Oct 11 10:54:33.603826 master-1 kubenswrapper[4771]: I1011 10:54:33.603782 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"placement-scripts" Oct 11 10:54:33.616505 master-1 kubenswrapper[4771]: I1011 10:54:33.616422 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/placement-6b597cbbf8-8j29d"] Oct 11 10:54:33.723284 master-1 kubenswrapper[4771]: I1011 10:54:33.723211 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-combined-ca-bundle\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.723605 master-1 kubenswrapper[4771]: I1011 10:54:33.723314 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-internal-tls-certs\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.723605 master-1 kubenswrapper[4771]: I1011 10:54:33.723347 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-public-tls-certs\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.723605 master-1 kubenswrapper[4771]: I1011 10:54:33.723405 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-config-data\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.723605 master-1 kubenswrapper[4771]: I1011 10:54:33.723441 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwr6t\" (UniqueName: \"kubernetes.io/projected/39f6e33d-5313-461d-ac81-59ab693324e8-kube-api-access-vwr6t\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.723605 master-1 kubenswrapper[4771]: I1011 10:54:33.723467 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-scripts\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.723605 master-1 kubenswrapper[4771]: I1011 10:54:33.723568 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/39f6e33d-5313-461d-ac81-59ab693324e8-logs\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.825033 master-1 kubenswrapper[4771]: I1011 10:54:33.824952 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-vwr6t\" (UniqueName: \"kubernetes.io/projected/39f6e33d-5313-461d-ac81-59ab693324e8-kube-api-access-vwr6t\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.825033 master-1 kubenswrapper[4771]: I1011 10:54:33.825039 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-scripts\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.825439 master-1 kubenswrapper[4771]: I1011 10:54:33.825105 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/39f6e33d-5313-461d-ac81-59ab693324e8-logs\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.825439 master-1 kubenswrapper[4771]: I1011 10:54:33.825141 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-combined-ca-bundle\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.825439 master-1 kubenswrapper[4771]: I1011 10:54:33.825207 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-internal-tls-certs\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.825439 master-1 kubenswrapper[4771]: I1011 10:54:33.825234 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-public-tls-certs\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.825439 master-1 kubenswrapper[4771]: I1011 10:54:33.825282 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-config-data\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.826546 master-1 kubenswrapper[4771]: I1011 10:54:33.826483 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/39f6e33d-5313-461d-ac81-59ab693324e8-logs\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.831330 master-1 kubenswrapper[4771]: I1011 10:54:33.831274 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-scripts\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.831694 master-1 kubenswrapper[4771]: I1011 10:54:33.831653 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-config-data\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.831745 master-1 kubenswrapper[4771]: I1011 10:54:33.831686 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-combined-ca-bundle\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.833332 master-1 kubenswrapper[4771]: I1011 10:54:33.833299 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-public-tls-certs\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.834462 master-1 kubenswrapper[4771]: I1011 10:54:33.834428 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/39f6e33d-5313-461d-ac81-59ab693324e8-internal-tls-certs\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.848907 master-1 kubenswrapper[4771]: I1011 10:54:33.848720 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwr6t\" (UniqueName: \"kubernetes.io/projected/39f6e33d-5313-461d-ac81-59ab693324e8-kube-api-access-vwr6t\") pod \"placement-6b597cbbf8-8j29d\" (UID: \"39f6e33d-5313-461d-ac81-59ab693324e8\") " pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:33.933503 master-1 kubenswrapper[4771]: I1011 10:54:33.932148 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:34.086082 master-1 kubenswrapper[4771]: I1011 10:54:34.085956 4771 generic.go:334] "Generic (PLEG): container finished" podID="30700706-219b-47c1-83cd-278584a3f182" containerID="2dcdb27cf0dbce506998b4c8cfe73f7847cd892689b076fcba313e976b8a5349" exitCode=0 Oct 11 10:54:34.086082 master-1 kubenswrapper[4771]: I1011 10:54:34.086038 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" event={"ID":"30700706-219b-47c1-83cd-278584a3f182","Type":"ContainerDied","Data":"2dcdb27cf0dbce506998b4c8cfe73f7847cd892689b076fcba313e976b8a5349"} Oct 11 10:54:36.746242 master-1 kubenswrapper[4771]: I1011 10:54:36.745980 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/keystone-848fcbb4df-cn592"] Oct 11 10:54:36.749205 master-1 kubenswrapper[4771]: I1011 10:54:36.749141 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.756484 master-1 kubenswrapper[4771]: I1011 10:54:36.755141 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-keystone-internal-svc" Oct 11 10:54:36.756484 master-1 kubenswrapper[4771]: I1011 10:54:36.755416 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-keystone-public-svc" Oct 11 10:54:36.756484 master-1 kubenswrapper[4771]: I1011 10:54:36.755555 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"keystone-config-data" Oct 11 10:54:36.757958 master-1 kubenswrapper[4771]: I1011 10:54:36.757929 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"keystone" Oct 11 10:54:36.758176 master-1 kubenswrapper[4771]: I1011 10:54:36.758154 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"keystone-scripts" Oct 11 10:54:36.759308 master-1 kubenswrapper[4771]: I1011 10:54:36.759259 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/keystone-848fcbb4df-cn592"] Oct 11 10:54:36.799255 master-1 kubenswrapper[4771]: I1011 10:54:36.799134 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-combined-ca-bundle\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.800503 master-1 kubenswrapper[4771]: I1011 10:54:36.800476 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-keys\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-credential-keys\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.800645 master-1 kubenswrapper[4771]: I1011 10:54:36.800631 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-config-data\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.800754 master-1 kubenswrapper[4771]: I1011 10:54:36.800741 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-scripts\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.800940 master-1 kubenswrapper[4771]: I1011 10:54:36.800923 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qxc\" (UniqueName: \"kubernetes.io/projected/0c78b078-6372-4692-8a56-d9aee58bffb8-kube-api-access-n6qxc\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.801453 master-1 kubenswrapper[4771]: I1011 10:54:36.801372 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-public-tls-certs\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.801453 master-1 kubenswrapper[4771]: I1011 10:54:36.801437 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-internal-tls-certs\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.801832 master-1 kubenswrapper[4771]: I1011 10:54:36.801802 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"fernet-keys\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-fernet-keys\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.904630 master-1 kubenswrapper[4771]: I1011 10:54:36.904560 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-combined-ca-bundle\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.905044 master-1 kubenswrapper[4771]: I1011 10:54:36.905024 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"credential-keys\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-credential-keys\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.905146 master-1 kubenswrapper[4771]: I1011 10:54:36.905132 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-config-data\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.905301 master-1 kubenswrapper[4771]: I1011 10:54:36.905284 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-scripts\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.905425 master-1 kubenswrapper[4771]: I1011 10:54:36.905409 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qxc\" (UniqueName: \"kubernetes.io/projected/0c78b078-6372-4692-8a56-d9aee58bffb8-kube-api-access-n6qxc\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.905596 master-1 kubenswrapper[4771]: I1011 10:54:36.905580 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-public-tls-certs\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.905697 master-1 kubenswrapper[4771]: I1011 10:54:36.905683 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-internal-tls-certs\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.905865 master-1 kubenswrapper[4771]: I1011 10:54:36.905847 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"fernet-keys\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-fernet-keys\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.909205 master-1 kubenswrapper[4771]: I1011 10:54:36.908962 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-combined-ca-bundle\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.909205 master-1 kubenswrapper[4771]: I1011 10:54:36.909153 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-scripts\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.909892 master-1 kubenswrapper[4771]: I1011 10:54:36.909847 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"credential-keys\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-credential-keys\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.910945 master-1 kubenswrapper[4771]: I1011 10:54:36.910783 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-internal-tls-certs\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.911069 master-1 kubenswrapper[4771]: I1011 10:54:36.911000 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-public-tls-certs\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.912755 master-1 kubenswrapper[4771]: I1011 10:54:36.912682 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"fernet-keys\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-fernet-keys\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.923620 master-1 kubenswrapper[4771]: I1011 10:54:36.914851 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/0c78b078-6372-4692-8a56-d9aee58bffb8-config-data\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:36.928250 master-1 kubenswrapper[4771]: I1011 10:54:36.928203 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qxc\" (UniqueName: \"kubernetes.io/projected/0c78b078-6372-4692-8a56-d9aee58bffb8-kube-api-access-n6qxc\") pod \"keystone-848fcbb4df-cn592\" (UID: \"0c78b078-6372-4692-8a56-d9aee58bffb8\") " pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:37.072458 master-1 kubenswrapper[4771]: I1011 10:54:37.072322 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:40.704488 master-1 kubenswrapper[4771]: I1011 10:54:40.704372 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:40.710992 master-1 kubenswrapper[4771]: I1011 10:54:40.710918 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:40.987136 master-1 kubenswrapper[4771]: I1011 10:54:40.984506 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:54:41.004098 master-1 kubenswrapper[4771]: I1011 10:54:41.004054 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-nb\") pod \"30700706-219b-47c1-83cd-278584a3f182\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " Oct 11 10:54:41.004270 master-1 kubenswrapper[4771]: I1011 10:54:41.004165 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-dns-svc\") pod \"30700706-219b-47c1-83cd-278584a3f182\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " Oct 11 10:54:41.004342 master-1 kubenswrapper[4771]: I1011 10:54:41.004322 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-sb\") pod \"30700706-219b-47c1-83cd-278584a3f182\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " Oct 11 10:54:41.004410 master-1 kubenswrapper[4771]: I1011 10:54:41.004344 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m5q5\" (UniqueName: \"kubernetes.io/projected/30700706-219b-47c1-83cd-278584a3f182-kube-api-access-2m5q5\") pod \"30700706-219b-47c1-83cd-278584a3f182\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " Oct 11 10:54:41.004410 master-1 kubenswrapper[4771]: I1011 10:54:41.004397 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-config\") pod \"30700706-219b-47c1-83cd-278584a3f182\" (UID: \"30700706-219b-47c1-83cd-278584a3f182\") " Oct 11 10:54:41.033295 master-1 kubenswrapper[4771]: I1011 10:54:41.033218 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30700706-219b-47c1-83cd-278584a3f182-kube-api-access-2m5q5" (OuterVolumeSpecName: "kube-api-access-2m5q5") pod "30700706-219b-47c1-83cd-278584a3f182" (UID: "30700706-219b-47c1-83cd-278584a3f182"). InnerVolumeSpecName "kube-api-access-2m5q5". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:54:41.056205 master-1 kubenswrapper[4771]: I1011 10:54:41.056132 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-config" (OuterVolumeSpecName: "config") pod "30700706-219b-47c1-83cd-278584a3f182" (UID: "30700706-219b-47c1-83cd-278584a3f182"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:41.061325 master-1 kubenswrapper[4771]: I1011 10:54:41.061185 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-sb" (OuterVolumeSpecName: "ovsdbserver-sb") pod "30700706-219b-47c1-83cd-278584a3f182" (UID: "30700706-219b-47c1-83cd-278584a3f182"). InnerVolumeSpecName "ovsdbserver-sb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:41.081629 master-1 kubenswrapper[4771]: I1011 10:54:41.081563 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "30700706-219b-47c1-83cd-278584a3f182" (UID: "30700706-219b-47c1-83cd-278584a3f182"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:41.087381 master-1 kubenswrapper[4771]: I1011 10:54:41.086827 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-nb" (OuterVolumeSpecName: "ovsdbserver-nb") pod "30700706-219b-47c1-83cd-278584a3f182" (UID: "30700706-219b-47c1-83cd-278584a3f182"). InnerVolumeSpecName "ovsdbserver-nb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:54:41.106451 master-1 kubenswrapper[4771]: I1011 10:54:41.106322 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:41.106451 master-1 kubenswrapper[4771]: I1011 10:54:41.106382 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-nb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:41.106451 master-1 kubenswrapper[4771]: I1011 10:54:41.106398 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:41.106451 master-1 kubenswrapper[4771]: I1011 10:54:41.106409 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/30700706-219b-47c1-83cd-278584a3f182-ovsdbserver-sb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:41.106451 master-1 kubenswrapper[4771]: I1011 10:54:41.106424 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-2m5q5\" (UniqueName: \"kubernetes.io/projected/30700706-219b-47c1-83cd-278584a3f182-kube-api-access-2m5q5\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:41.143441 master-1 kubenswrapper[4771]: I1011 10:54:41.143386 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" event={"ID":"30700706-219b-47c1-83cd-278584a3f182","Type":"ContainerDied","Data":"cc8ace77e3138b8e3e45d04fc9090a4ecc543f54342ec2f309cdbc89855a76b5"} Oct 11 10:54:41.143441 master-1 kubenswrapper[4771]: I1011 10:54:41.143443 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" Oct 11 10:54:41.143679 master-1 kubenswrapper[4771]: I1011 10:54:41.143486 4771 scope.go:117] "RemoveContainer" containerID="2dcdb27cf0dbce506998b4c8cfe73f7847cd892689b076fcba313e976b8a5349" Oct 11 10:54:41.147768 master-1 kubenswrapper[4771]: I1011 10:54:41.147665 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerStarted","Data":"b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd"} Oct 11 10:54:41.153077 master-1 kubenswrapper[4771]: I1011 10:54:41.153033 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/prometheus-metric-storage-0" Oct 11 10:54:41.191974 master-1 kubenswrapper[4771]: I1011 10:54:41.191938 4771 scope.go:117] "RemoveContainer" containerID="06ae06abec101801ffcb11de5d066d694be4874cbd2110b56c80026a91417fc8" Oct 11 10:54:41.232944 master-1 kubenswrapper[4771]: I1011 10:54:41.232881 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-6c99f4877f-vjhdt"] Oct 11 10:54:41.237114 master-1 kubenswrapper[4771]: I1011 10:54:41.237056 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-6c99f4877f-vjhdt"] Oct 11 10:54:41.337765 master-1 kubenswrapper[4771]: I1011 10:54:41.337705 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/keystone-848fcbb4df-cn592"] Oct 11 10:54:41.340549 master-1 kubenswrapper[4771]: W1011 10:54:41.340481 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c78b078_6372_4692_8a56_d9aee58bffb8.slice/crio-349a33e83700cb100d3c7dfe2d13003eb2f27916ff1d46e4fac294cad84acdcc WatchSource:0}: Error finding container 349a33e83700cb100d3c7dfe2d13003eb2f27916ff1d46e4fac294cad84acdcc: Status 404 returned error can't find the container with id 349a33e83700cb100d3c7dfe2d13003eb2f27916ff1d46e4fac294cad84acdcc Oct 11 10:54:41.442604 master-1 kubenswrapper[4771]: W1011 10:54:41.442540 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39f6e33d_5313_461d_ac81_59ab693324e8.slice/crio-c0b23cbb30962b0597a0653b7895f57534e0520c3a6e69c040624aa8e0977731 WatchSource:0}: Error finding container c0b23cbb30962b0597a0653b7895f57534e0520c3a6e69c040624aa8e0977731: Status 404 returned error can't find the container with id c0b23cbb30962b0597a0653b7895f57534e0520c3a6e69c040624aa8e0977731 Oct 11 10:54:41.443258 master-1 kubenswrapper[4771]: I1011 10:54:41.442862 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/placement-6b597cbbf8-8j29d"] Oct 11 10:54:42.177725 master-1 kubenswrapper[4771]: I1011 10:54:42.177627 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/placement-6b597cbbf8-8j29d" event={"ID":"39f6e33d-5313-461d-ac81-59ab693324e8","Type":"ContainerStarted","Data":"c0b23cbb30962b0597a0653b7895f57534e0520c3a6e69c040624aa8e0977731"} Oct 11 10:54:42.183374 master-1 kubenswrapper[4771]: I1011 10:54:42.183277 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"499f9e94-a738-484d-ae4b-0cc221750d1c","Type":"ContainerStarted","Data":"2d4fd9e07f37d7d0e4c5b7147d47642c209dd291fd8ea33730298efb1acb5aa4"} Oct 11 10:54:42.191933 master-1 kubenswrapper[4771]: I1011 10:54:42.191776 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/keystone-848fcbb4df-cn592" event={"ID":"0c78b078-6372-4692-8a56-d9aee58bffb8","Type":"ContainerStarted","Data":"349a33e83700cb100d3c7dfe2d13003eb2f27916ff1d46e4fac294cad84acdcc"} Oct 11 10:54:42.199974 master-1 kubenswrapper[4771]: I1011 10:54:42.199909 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"18861a21-406e-479b-8712-9a62ca2ebf4a","Type":"ContainerStarted","Data":"497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b"} Oct 11 10:54:42.448616 master-1 kubenswrapper[4771]: I1011 10:54:42.447437 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30700706-219b-47c1-83cd-278584a3f182" path="/var/lib/kubelet/pods/30700706-219b-47c1-83cd-278584a3f182/volumes" Oct 11 10:54:42.948823 master-1 kubenswrapper[4771]: I1011 10:54:42.948722 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/dnsmasq-dns-6c99f4877f-vjhdt" podUID="30700706-219b-47c1-83cd-278584a3f182" containerName="dnsmasq-dns" probeResult="failure" output="dial tcp 10.129.0.120:5353: i/o timeout" Oct 11 10:54:43.215948 master-1 kubenswrapper[4771]: I1011 10:54:43.215872 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"499f9e94-a738-484d-ae4b-0cc221750d1c","Type":"ContainerStarted","Data":"b479c48e028ed10f47dcf8ff360fd70182a69875e2d6e7028a9c345aed74bb52"} Oct 11 10:54:43.222136 master-1 kubenswrapper[4771]: I1011 10:54:43.222054 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"18861a21-406e-479b-8712-9a62ca2ebf4a","Type":"ContainerStarted","Data":"122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f"} Oct 11 10:54:43.254679 master-1 kubenswrapper[4771]: I1011 10:54:43.254546 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/glance-b5802-default-internal-api-1" podStartSLOduration=10.033035491 podStartE2EDuration="21.254502634s" podCreationTimestamp="2025-10-11 10:54:22 +0000 UTC" firstStartedPulling="2025-10-11 10:54:29.667669302 +0000 UTC m=+1701.641895743" lastFinishedPulling="2025-10-11 10:54:40.889136425 +0000 UTC m=+1712.863362886" observedRunningTime="2025-10-11 10:54:43.243755253 +0000 UTC m=+1715.217981704" watchObservedRunningTime="2025-10-11 10:54:43.254502634 +0000 UTC m=+1715.228729075" Oct 11 10:54:43.291974 master-1 kubenswrapper[4771]: I1011 10:54:43.291892 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/glance-b5802-default-external-api-0" podStartSLOduration=8.598630598 podStartE2EDuration="21.291868104s" podCreationTimestamp="2025-10-11 10:54:22 +0000 UTC" firstStartedPulling="2025-10-11 10:54:28.177681193 +0000 UTC m=+1700.151907654" lastFinishedPulling="2025-10-11 10:54:40.870918689 +0000 UTC m=+1712.845145160" observedRunningTime="2025-10-11 10:54:43.289387592 +0000 UTC m=+1715.263614043" watchObservedRunningTime="2025-10-11 10:54:43.291868104 +0000 UTC m=+1715.266094545" Oct 11 10:54:47.193189 master-1 kubenswrapper[4771]: I1011 10:54:47.193096 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:47.193189 master-1 kubenswrapper[4771]: I1011 10:54:47.193176 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:47.223779 master-1 kubenswrapper[4771]: I1011 10:54:47.223655 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:47.248154 master-1 kubenswrapper[4771]: I1011 10:54:47.248096 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:47.257148 master-1 kubenswrapper[4771]: I1011 10:54:47.257118 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:47.257148 master-1 kubenswrapper[4771]: I1011 10:54:47.257154 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:48.767220 master-1 kubenswrapper[4771]: I1011 10:54:48.767096 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:48.767220 master-1 kubenswrapper[4771]: I1011 10:54:48.767176 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:48.803765 master-1 kubenswrapper[4771]: I1011 10:54:48.803663 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:48.832304 master-1 kubenswrapper[4771]: I1011 10:54:48.832190 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:49.291029 master-1 kubenswrapper[4771]: I1011 10:54:49.290962 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:49.291631 master-1 kubenswrapper[4771]: I1011 10:54:49.291554 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:49.536339 master-1 kubenswrapper[4771]: I1011 10:54:49.536252 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:49.536696 master-1 kubenswrapper[4771]: I1011 10:54:49.536558 4771 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Oct 11 10:54:49.541658 master-1 kubenswrapper[4771]: I1011 10:54:49.541599 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:54:50.291304 master-1 kubenswrapper[4771]: I1011 10:54:50.291177 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerStarted","Data":"1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624"} Oct 11 10:54:50.292225 master-1 kubenswrapper[4771]: I1011 10:54:50.291416 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="ceilometer-central-agent" containerID="cri-o://5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027" gracePeriod=30 Oct 11 10:54:50.292225 master-1 kubenswrapper[4771]: I1011 10:54:50.291490 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="ceilometer-notification-agent" containerID="cri-o://9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb" gracePeriod=30 Oct 11 10:54:50.292225 master-1 kubenswrapper[4771]: I1011 10:54:50.291503 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="sg-core" containerID="cri-o://b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd" gracePeriod=30 Oct 11 10:54:50.292225 master-1 kubenswrapper[4771]: I1011 10:54:50.291426 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 10:54:50.292225 master-1 kubenswrapper[4771]: I1011 10:54:50.291739 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="proxy-httpd" containerID="cri-o://1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624" gracePeriod=30 Oct 11 10:54:50.297191 master-1 kubenswrapper[4771]: I1011 10:54:50.297115 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/keystone-848fcbb4df-cn592" event={"ID":"0c78b078-6372-4692-8a56-d9aee58bffb8","Type":"ContainerStarted","Data":"0a7ad84194d539ff6b8c5be28c4477a9fa419f7f96949c0f702b34e73d791b55"} Oct 11 10:54:50.297465 master-1 kubenswrapper[4771]: I1011 10:54:50.297387 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:54:50.299794 master-1 kubenswrapper[4771]: I1011 10:54:50.299740 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/placement-6b597cbbf8-8j29d" event={"ID":"39f6e33d-5313-461d-ac81-59ab693324e8","Type":"ContainerStarted","Data":"b5b0b9229993058400a57de708bc63e8c4ccaf9117579a7fc4d78a9e6a15bb9c"} Oct 11 10:54:50.299899 master-1 kubenswrapper[4771]: I1011 10:54:50.299791 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/placement-6b597cbbf8-8j29d" event={"ID":"39f6e33d-5313-461d-ac81-59ab693324e8","Type":"ContainerStarted","Data":"dbc539ad66fd3494364c09c43de44cc8278a2d6c3cb528609ab332a556e7d475"} Oct 11 10:54:50.499534 master-1 kubenswrapper[4771]: I1011 10:54:50.499420 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=2.620208204 podStartE2EDuration="28.499392856s" podCreationTimestamp="2025-10-11 10:54:22 +0000 UTC" firstStartedPulling="2025-10-11 10:54:23.281867057 +0000 UTC m=+1695.256093498" lastFinishedPulling="2025-10-11 10:54:49.161051709 +0000 UTC m=+1721.135278150" observedRunningTime="2025-10-11 10:54:50.488203453 +0000 UTC m=+1722.462429934" watchObservedRunningTime="2025-10-11 10:54:50.499392856 +0000 UTC m=+1722.473619337" Oct 11 10:54:50.522080 master-1 kubenswrapper[4771]: I1011 10:54:50.521941 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/placement-6b597cbbf8-8j29d" podStartSLOduration=9.841981992000001 podStartE2EDuration="17.521900337s" podCreationTimestamp="2025-10-11 10:54:33 +0000 UTC" firstStartedPulling="2025-10-11 10:54:41.449069657 +0000 UTC m=+1713.423296098" lastFinishedPulling="2025-10-11 10:54:49.128988002 +0000 UTC m=+1721.103214443" observedRunningTime="2025-10-11 10:54:50.517052737 +0000 UTC m=+1722.491279198" watchObservedRunningTime="2025-10-11 10:54:50.521900337 +0000 UTC m=+1722.496126788" Oct 11 10:54:50.630333 master-1 kubenswrapper[4771]: I1011 10:54:50.630181 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/keystone-848fcbb4df-cn592" podStartSLOduration=6.842980011 podStartE2EDuration="14.630153325s" podCreationTimestamp="2025-10-11 10:54:36 +0000 UTC" firstStartedPulling="2025-10-11 10:54:41.345029431 +0000 UTC m=+1713.319255872" lastFinishedPulling="2025-10-11 10:54:49.132202745 +0000 UTC m=+1721.106429186" observedRunningTime="2025-10-11 10:54:50.622586056 +0000 UTC m=+1722.596812517" watchObservedRunningTime="2025-10-11 10:54:50.630153325 +0000 UTC m=+1722.604379806" Oct 11 10:54:51.312337 master-1 kubenswrapper[4771]: I1011 10:54:51.312256 4771 generic.go:334] "Generic (PLEG): container finished" podID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerID="1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624" exitCode=0 Oct 11 10:54:51.312337 master-1 kubenswrapper[4771]: I1011 10:54:51.312306 4771 generic.go:334] "Generic (PLEG): container finished" podID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerID="b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd" exitCode=2 Oct 11 10:54:51.312337 master-1 kubenswrapper[4771]: I1011 10:54:51.312313 4771 generic.go:334] "Generic (PLEG): container finished" podID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerID="5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027" exitCode=0 Oct 11 10:54:51.313342 master-1 kubenswrapper[4771]: I1011 10:54:51.313304 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerDied","Data":"1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624"} Oct 11 10:54:51.313409 master-1 kubenswrapper[4771]: I1011 10:54:51.313366 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerDied","Data":"b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd"} Oct 11 10:54:51.313409 master-1 kubenswrapper[4771]: I1011 10:54:51.313379 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerDied","Data":"5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027"} Oct 11 10:54:51.313886 master-1 kubenswrapper[4771]: I1011 10:54:51.313855 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:51.313931 master-1 kubenswrapper[4771]: I1011 10:54:51.313891 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:54:51.482334 master-1 kubenswrapper[4771]: I1011 10:54:51.482253 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:51.482668 master-1 kubenswrapper[4771]: I1011 10:54:51.482423 4771 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Oct 11 10:54:51.487376 master-1 kubenswrapper[4771]: I1011 10:54:51.486240 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:53.257313 master-1 kubenswrapper[4771]: I1011 10:54:53.257213 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:54:53.333296 master-1 kubenswrapper[4771]: I1011 10:54:53.333196 4771 generic.go:334] "Generic (PLEG): container finished" podID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerID="9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb" exitCode=0 Oct 11 10:54:53.333296 master-1 kubenswrapper[4771]: I1011 10:54:53.333304 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:54:53.333765 master-1 kubenswrapper[4771]: I1011 10:54:53.333389 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerDied","Data":"9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb"} Oct 11 10:54:53.333765 master-1 kubenswrapper[4771]: I1011 10:54:53.333554 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"6508f17e-afc7-44dd-89b4-2efa8a124b12","Type":"ContainerDied","Data":"7c53a7f6f217d02a6155b61317bdcecef01a919dcbe718b5d7a4a4096ceec2ae"} Oct 11 10:54:53.333765 master-1 kubenswrapper[4771]: I1011 10:54:53.333593 4771 scope.go:117] "RemoveContainer" containerID="1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624" Oct 11 10:54:53.374276 master-1 kubenswrapper[4771]: I1011 10:54:53.371717 4771 scope.go:117] "RemoveContainer" containerID="b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd" Oct 11 10:54:53.412389 master-1 kubenswrapper[4771]: I1011 10:54:53.412277 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-config-data\") pod \"6508f17e-afc7-44dd-89b4-2efa8a124b12\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " Oct 11 10:54:53.412767 master-1 kubenswrapper[4771]: I1011 10:54:53.412424 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-run-httpd\") pod \"6508f17e-afc7-44dd-89b4-2efa8a124b12\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " Oct 11 10:54:53.412767 master-1 kubenswrapper[4771]: I1011 10:54:53.412501 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-sg-core-conf-yaml\") pod \"6508f17e-afc7-44dd-89b4-2efa8a124b12\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " Oct 11 10:54:53.412767 master-1 kubenswrapper[4771]: I1011 10:54:53.412660 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-log-httpd\") pod \"6508f17e-afc7-44dd-89b4-2efa8a124b12\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " Oct 11 10:54:53.412921 master-1 kubenswrapper[4771]: I1011 10:54:53.412863 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "6508f17e-afc7-44dd-89b4-2efa8a124b12" (UID: "6508f17e-afc7-44dd-89b4-2efa8a124b12"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:54:53.413124 master-1 kubenswrapper[4771]: I1011 10:54:53.413085 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "6508f17e-afc7-44dd-89b4-2efa8a124b12" (UID: "6508f17e-afc7-44dd-89b4-2efa8a124b12"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:54:53.413466 master-1 kubenswrapper[4771]: I1011 10:54:53.413417 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-combined-ca-bundle\") pod \"6508f17e-afc7-44dd-89b4-2efa8a124b12\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " Oct 11 10:54:53.413718 master-1 kubenswrapper[4771]: I1011 10:54:53.413615 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76sj2\" (UniqueName: \"kubernetes.io/projected/6508f17e-afc7-44dd-89b4-2efa8a124b12-kube-api-access-76sj2\") pod \"6508f17e-afc7-44dd-89b4-2efa8a124b12\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " Oct 11 10:54:53.413939 master-1 kubenswrapper[4771]: I1011 10:54:53.413902 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-scripts\") pod \"6508f17e-afc7-44dd-89b4-2efa8a124b12\" (UID: \"6508f17e-afc7-44dd-89b4-2efa8a124b12\") " Oct 11 10:54:53.414912 master-1 kubenswrapper[4771]: I1011 10:54:53.414876 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:53.414912 master-1 kubenswrapper[4771]: I1011 10:54:53.414911 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/6508f17e-afc7-44dd-89b4-2efa8a124b12-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:53.416748 master-1 kubenswrapper[4771]: I1011 10:54:53.416715 4771 scope.go:117] "RemoveContainer" containerID="9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb" Oct 11 10:54:53.416996 master-1 kubenswrapper[4771]: I1011 10:54:53.416925 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6508f17e-afc7-44dd-89b4-2efa8a124b12-kube-api-access-76sj2" (OuterVolumeSpecName: "kube-api-access-76sj2") pod "6508f17e-afc7-44dd-89b4-2efa8a124b12" (UID: "6508f17e-afc7-44dd-89b4-2efa8a124b12"). InnerVolumeSpecName "kube-api-access-76sj2". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:54:53.417880 master-1 kubenswrapper[4771]: I1011 10:54:53.417817 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-scripts" (OuterVolumeSpecName: "scripts") pod "6508f17e-afc7-44dd-89b4-2efa8a124b12" (UID: "6508f17e-afc7-44dd-89b4-2efa8a124b12"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:53.436974 master-1 kubenswrapper[4771]: I1011 10:54:53.436817 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "6508f17e-afc7-44dd-89b4-2efa8a124b12" (UID: "6508f17e-afc7-44dd-89b4-2efa8a124b12"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:53.488086 master-1 kubenswrapper[4771]: I1011 10:54:53.488000 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "6508f17e-afc7-44dd-89b4-2efa8a124b12" (UID: "6508f17e-afc7-44dd-89b4-2efa8a124b12"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:53.496192 master-1 kubenswrapper[4771]: I1011 10:54:53.496135 4771 scope.go:117] "RemoveContainer" containerID="5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027" Oct 11 10:54:53.513166 master-1 kubenswrapper[4771]: I1011 10:54:53.513121 4771 scope.go:117] "RemoveContainer" containerID="1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624" Oct 11 10:54:53.513829 master-1 kubenswrapper[4771]: E1011 10:54:53.513779 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624\": container with ID starting with 1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624 not found: ID does not exist" containerID="1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624" Oct 11 10:54:53.513885 master-1 kubenswrapper[4771]: I1011 10:54:53.513845 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624"} err="failed to get container status \"1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624\": rpc error: code = NotFound desc = could not find container \"1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624\": container with ID starting with 1959dfa42cdae287607dba414d321cf7399ac93af3a551ad5d6c73607087a624 not found: ID does not exist" Oct 11 10:54:53.513933 master-1 kubenswrapper[4771]: I1011 10:54:53.513902 4771 scope.go:117] "RemoveContainer" containerID="b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd" Oct 11 10:54:53.514472 master-1 kubenswrapper[4771]: E1011 10:54:53.514441 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd\": container with ID starting with b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd not found: ID does not exist" containerID="b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd" Oct 11 10:54:53.514533 master-1 kubenswrapper[4771]: I1011 10:54:53.514467 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd"} err="failed to get container status \"b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd\": rpc error: code = NotFound desc = could not find container \"b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd\": container with ID starting with b88255ee4532787c4c00c2711a51a51676d239e5c1a4bb061ac2e5c547c75ccd not found: ID does not exist" Oct 11 10:54:53.514533 master-1 kubenswrapper[4771]: I1011 10:54:53.514485 4771 scope.go:117] "RemoveContainer" containerID="9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb" Oct 11 10:54:53.514794 master-1 kubenswrapper[4771]: E1011 10:54:53.514767 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb\": container with ID starting with 9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb not found: ID does not exist" containerID="9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb" Oct 11 10:54:53.514851 master-1 kubenswrapper[4771]: I1011 10:54:53.514792 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb"} err="failed to get container status \"9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb\": rpc error: code = NotFound desc = could not find container \"9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb\": container with ID starting with 9ae9b7cbf86ceb270193377537f9fbc6af1ee710bbbb22c0a85e001eae30ccdb not found: ID does not exist" Oct 11 10:54:53.514851 master-1 kubenswrapper[4771]: I1011 10:54:53.514808 4771 scope.go:117] "RemoveContainer" containerID="5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027" Oct 11 10:54:53.515103 master-1 kubenswrapper[4771]: E1011 10:54:53.515075 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027\": container with ID starting with 5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027 not found: ID does not exist" containerID="5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027" Oct 11 10:54:53.515157 master-1 kubenswrapper[4771]: I1011 10:54:53.515097 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027"} err="failed to get container status \"5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027\": rpc error: code = NotFound desc = could not find container \"5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027\": container with ID starting with 5b743e4068a6cb14bb814fdd41d8cf6bd84e84d0f5ad68d1776ee95613452027 not found: ID does not exist" Oct 11 10:54:53.516807 master-1 kubenswrapper[4771]: I1011 10:54:53.516753 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:53.516807 master-1 kubenswrapper[4771]: I1011 10:54:53.516793 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-76sj2\" (UniqueName: \"kubernetes.io/projected/6508f17e-afc7-44dd-89b4-2efa8a124b12-kube-api-access-76sj2\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:53.516807 master-1 kubenswrapper[4771]: I1011 10:54:53.516807 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:53.516931 master-1 kubenswrapper[4771]: I1011 10:54:53.516823 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:53.532712 master-1 kubenswrapper[4771]: I1011 10:54:53.532630 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-config-data" (OuterVolumeSpecName: "config-data") pod "6508f17e-afc7-44dd-89b4-2efa8a124b12" (UID: "6508f17e-afc7-44dd-89b4-2efa8a124b12"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:53.618902 master-1 kubenswrapper[4771]: I1011 10:54:53.618813 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/6508f17e-afc7-44dd-89b4-2efa8a124b12-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:53.676919 master-1 kubenswrapper[4771]: I1011 10:54:53.676809 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:54:53.682526 master-1 kubenswrapper[4771]: I1011 10:54:53.682465 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:54:53.722022 master-1 kubenswrapper[4771]: I1011 10:54:53.721799 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:54:53.722797 master-1 kubenswrapper[4771]: E1011 10:54:53.722777 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="sg-core" Oct 11 10:54:53.722879 master-1 kubenswrapper[4771]: I1011 10:54:53.722868 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="sg-core" Oct 11 10:54:53.722957 master-1 kubenswrapper[4771]: E1011 10:54:53.722947 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="proxy-httpd" Oct 11 10:54:53.723010 master-1 kubenswrapper[4771]: I1011 10:54:53.723001 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="proxy-httpd" Oct 11 10:54:53.724207 master-1 kubenswrapper[4771]: E1011 10:54:53.724193 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="ceilometer-notification-agent" Oct 11 10:54:53.724295 master-1 kubenswrapper[4771]: I1011 10:54:53.724285 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="ceilometer-notification-agent" Oct 11 10:54:53.724399 master-1 kubenswrapper[4771]: E1011 10:54:53.724385 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="30700706-219b-47c1-83cd-278584a3f182" containerName="dnsmasq-dns" Oct 11 10:54:53.724465 master-1 kubenswrapper[4771]: I1011 10:54:53.724455 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="30700706-219b-47c1-83cd-278584a3f182" containerName="dnsmasq-dns" Oct 11 10:54:53.724539 master-1 kubenswrapper[4771]: E1011 10:54:53.724529 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="ceilometer-central-agent" Oct 11 10:54:53.724594 master-1 kubenswrapper[4771]: I1011 10:54:53.724585 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="ceilometer-central-agent" Oct 11 10:54:53.724664 master-1 kubenswrapper[4771]: E1011 10:54:53.724655 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="30700706-219b-47c1-83cd-278584a3f182" containerName="init" Oct 11 10:54:53.724718 master-1 kubenswrapper[4771]: I1011 10:54:53.724709 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="30700706-219b-47c1-83cd-278584a3f182" containerName="init" Oct 11 10:54:53.724954 master-1 kubenswrapper[4771]: I1011 10:54:53.724939 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="ceilometer-notification-agent" Oct 11 10:54:53.725029 master-1 kubenswrapper[4771]: I1011 10:54:53.725018 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="proxy-httpd" Oct 11 10:54:53.725095 master-1 kubenswrapper[4771]: I1011 10:54:53.725085 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="ceilometer-central-agent" Oct 11 10:54:53.725175 master-1 kubenswrapper[4771]: I1011 10:54:53.725163 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="30700706-219b-47c1-83cd-278584a3f182" containerName="dnsmasq-dns" Oct 11 10:54:53.725242 master-1 kubenswrapper[4771]: I1011 10:54:53.725232 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" containerName="sg-core" Oct 11 10:54:53.727638 master-1 kubenswrapper[4771]: I1011 10:54:53.727607 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:54:53.731375 master-1 kubenswrapper[4771]: I1011 10:54:53.731306 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 10:54:53.731530 master-1 kubenswrapper[4771]: I1011 10:54:53.731474 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 10:54:53.743769 master-1 kubenswrapper[4771]: I1011 10:54:53.741495 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:54:53.823771 master-1 kubenswrapper[4771]: I1011 10:54:53.823688 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.823771 master-1 kubenswrapper[4771]: I1011 10:54:53.823790 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-run-httpd\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.824161 master-1 kubenswrapper[4771]: I1011 10:54:53.823982 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-config-data\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.824199 master-1 kubenswrapper[4771]: I1011 10:54:53.824160 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj45w\" (UniqueName: \"kubernetes.io/projected/49ec9c51-e085-4cfa-8ce7-387a02f23731-kube-api-access-rj45w\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.824245 master-1 kubenswrapper[4771]: I1011 10:54:53.824208 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-scripts\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.824600 master-1 kubenswrapper[4771]: I1011 10:54:53.824324 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-log-httpd\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.824600 master-1 kubenswrapper[4771]: I1011 10:54:53.824441 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.927066 master-1 kubenswrapper[4771]: I1011 10:54:53.926973 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-run-httpd\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.927398 master-1 kubenswrapper[4771]: I1011 10:54:53.927098 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-config-data\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.927398 master-1 kubenswrapper[4771]: I1011 10:54:53.927146 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-rj45w\" (UniqueName: \"kubernetes.io/projected/49ec9c51-e085-4cfa-8ce7-387a02f23731-kube-api-access-rj45w\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.927398 master-1 kubenswrapper[4771]: I1011 10:54:53.927171 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-scripts\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.927398 master-1 kubenswrapper[4771]: I1011 10:54:53.927200 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-log-httpd\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.927398 master-1 kubenswrapper[4771]: I1011 10:54:53.927231 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.927398 master-1 kubenswrapper[4771]: I1011 10:54:53.927283 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.927845 master-1 kubenswrapper[4771]: I1011 10:54:53.927775 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-run-httpd\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.927959 master-1 kubenswrapper[4771]: I1011 10:54:53.927906 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-log-httpd\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.931554 master-1 kubenswrapper[4771]: I1011 10:54:53.931508 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.932696 master-1 kubenswrapper[4771]: I1011 10:54:53.932627 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-config-data\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.933445 master-1 kubenswrapper[4771]: I1011 10:54:53.933388 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-scripts\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.935743 master-1 kubenswrapper[4771]: I1011 10:54:53.935705 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:53.946812 master-1 kubenswrapper[4771]: I1011 10:54:53.946745 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj45w\" (UniqueName: \"kubernetes.io/projected/49ec9c51-e085-4cfa-8ce7-387a02f23731-kube-api-access-rj45w\") pod \"ceilometer-0\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " pod="openstack/ceilometer-0" Oct 11 10:54:54.057314 master-1 kubenswrapper[4771]: I1011 10:54:54.057217 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:54:54.453803 master-1 kubenswrapper[4771]: I1011 10:54:54.453591 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6508f17e-afc7-44dd-89b4-2efa8a124b12" path="/var/lib/kubelet/pods/6508f17e-afc7-44dd-89b4-2efa8a124b12/volumes" Oct 11 10:54:54.591601 master-1 kubenswrapper[4771]: I1011 10:54:54.591439 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:54:54.600186 master-1 kubenswrapper[4771]: W1011 10:54:54.600102 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49ec9c51_e085_4cfa_8ce7_387a02f23731.slice/crio-ada9da27885a5797d5d7044aa85caf6f5ca08e4448276ab5e54d26349906f001 WatchSource:0}: Error finding container ada9da27885a5797d5d7044aa85caf6f5ca08e4448276ab5e54d26349906f001: Status 404 returned error can't find the container with id ada9da27885a5797d5d7044aa85caf6f5ca08e4448276ab5e54d26349906f001 Oct 11 10:54:55.161268 master-1 kubenswrapper[4771]: I1011 10:54:55.160182 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:54:55.162885 master-1 kubenswrapper[4771]: I1011 10:54:55.161459 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/glance-b5802-default-internal-api-1" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-log" containerID="cri-o://2d4fd9e07f37d7d0e4c5b7147d47642c209dd291fd8ea33730298efb1acb5aa4" gracePeriod=30 Oct 11 10:54:55.162885 master-1 kubenswrapper[4771]: I1011 10:54:55.161665 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/glance-b5802-default-internal-api-1" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-httpd" containerID="cri-o://b479c48e028ed10f47dcf8ff360fd70182a69875e2d6e7028a9c345aed74bb52" gracePeriod=30 Oct 11 10:54:55.358682 master-1 kubenswrapper[4771]: I1011 10:54:55.358497 4771 generic.go:334] "Generic (PLEG): container finished" podID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerID="2d4fd9e07f37d7d0e4c5b7147d47642c209dd291fd8ea33730298efb1acb5aa4" exitCode=143 Oct 11 10:54:55.358682 master-1 kubenswrapper[4771]: I1011 10:54:55.358544 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"499f9e94-a738-484d-ae4b-0cc221750d1c","Type":"ContainerDied","Data":"2d4fd9e07f37d7d0e4c5b7147d47642c209dd291fd8ea33730298efb1acb5aa4"} Oct 11 10:54:55.362172 master-1 kubenswrapper[4771]: I1011 10:54:55.362122 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerStarted","Data":"1c3861fe4b88a03a7f5a37466aa7554573bbeb56e12a609a086b0d7cf9119e59"} Oct 11 10:54:55.362172 master-1 kubenswrapper[4771]: I1011 10:54:55.362173 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerStarted","Data":"ada9da27885a5797d5d7044aa85caf6f5ca08e4448276ab5e54d26349906f001"} Oct 11 10:54:56.377097 master-1 kubenswrapper[4771]: I1011 10:54:56.377017 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerStarted","Data":"0581a657f8cf01879d33e71f4db4cc2df261f4f45ead619016173a151ac38bcc"} Oct 11 10:54:57.413263 master-1 kubenswrapper[4771]: I1011 10:54:57.412836 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerStarted","Data":"e1413b97fa6430a651152f91df8a84c4a32dbe4f3d81aabb8fb9fea0809e7a16"} Oct 11 10:54:58.426095 master-1 kubenswrapper[4771]: I1011 10:54:58.426026 4771 generic.go:334] "Generic (PLEG): container finished" podID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerID="b479c48e028ed10f47dcf8ff360fd70182a69875e2d6e7028a9c345aed74bb52" exitCode=0 Oct 11 10:54:58.426970 master-1 kubenswrapper[4771]: I1011 10:54:58.426287 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"499f9e94-a738-484d-ae4b-0cc221750d1c","Type":"ContainerDied","Data":"b479c48e028ed10f47dcf8ff360fd70182a69875e2d6e7028a9c345aed74bb52"} Oct 11 10:54:58.431806 master-1 kubenswrapper[4771]: I1011 10:54:58.431736 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerStarted","Data":"3b0917f1e0c562a330c2d467c28336993eadaec7d16c4d43d62cfb2b0ba25b4b"} Oct 11 10:54:58.476116 master-1 kubenswrapper[4771]: I1011 10:54:58.475757 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=2.276729548 podStartE2EDuration="5.475623794s" podCreationTimestamp="2025-10-11 10:54:53 +0000 UTC" firstStartedPulling="2025-10-11 10:54:54.60689416 +0000 UTC m=+1726.581120641" lastFinishedPulling="2025-10-11 10:54:57.805788446 +0000 UTC m=+1729.780014887" observedRunningTime="2025-10-11 10:54:58.47343399 +0000 UTC m=+1730.447660481" watchObservedRunningTime="2025-10-11 10:54:58.475623794 +0000 UTC m=+1730.449850255" Oct 11 10:54:58.988390 master-1 kubenswrapper[4771]: I1011 10:54:58.988303 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.161844 master-1 kubenswrapper[4771]: I1011 10:54:59.161204 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-config-data\") pod \"499f9e94-a738-484d-ae4b-0cc221750d1c\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " Oct 11 10:54:59.161844 master-1 kubenswrapper[4771]: I1011 10:54:59.161617 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"glance\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"499f9e94-a738-484d-ae4b-0cc221750d1c\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " Oct 11 10:54:59.161844 master-1 kubenswrapper[4771]: I1011 10:54:59.161659 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-httpd-run\") pod \"499f9e94-a738-484d-ae4b-0cc221750d1c\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " Oct 11 10:54:59.161844 master-1 kubenswrapper[4771]: I1011 10:54:59.161730 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-logs\") pod \"499f9e94-a738-484d-ae4b-0cc221750d1c\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " Oct 11 10:54:59.161844 master-1 kubenswrapper[4771]: I1011 10:54:59.161816 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5hg2\" (UniqueName: \"kubernetes.io/projected/499f9e94-a738-484d-ae4b-0cc221750d1c-kube-api-access-k5hg2\") pod \"499f9e94-a738-484d-ae4b-0cc221750d1c\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " Oct 11 10:54:59.162494 master-1 kubenswrapper[4771]: I1011 10:54:59.161891 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-internal-tls-certs\") pod \"499f9e94-a738-484d-ae4b-0cc221750d1c\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " Oct 11 10:54:59.162494 master-1 kubenswrapper[4771]: I1011 10:54:59.161914 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-combined-ca-bundle\") pod \"499f9e94-a738-484d-ae4b-0cc221750d1c\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " Oct 11 10:54:59.162494 master-1 kubenswrapper[4771]: I1011 10:54:59.161950 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-scripts\") pod \"499f9e94-a738-484d-ae4b-0cc221750d1c\" (UID: \"499f9e94-a738-484d-ae4b-0cc221750d1c\") " Oct 11 10:54:59.162494 master-1 kubenswrapper[4771]: I1011 10:54:59.161983 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-httpd-run" (OuterVolumeSpecName: "httpd-run") pod "499f9e94-a738-484d-ae4b-0cc221750d1c" (UID: "499f9e94-a738-484d-ae4b-0cc221750d1c"). InnerVolumeSpecName "httpd-run". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:54:59.162494 master-1 kubenswrapper[4771]: I1011 10:54:59.162405 4771 reconciler_common.go:293] "Volume detached for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-httpd-run\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:59.163056 master-1 kubenswrapper[4771]: I1011 10:54:59.162991 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-logs" (OuterVolumeSpecName: "logs") pod "499f9e94-a738-484d-ae4b-0cc221750d1c" (UID: "499f9e94-a738-484d-ae4b-0cc221750d1c"). InnerVolumeSpecName "logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:54:59.166949 master-1 kubenswrapper[4771]: I1011 10:54:59.166861 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-scripts" (OuterVolumeSpecName: "scripts") pod "499f9e94-a738-484d-ae4b-0cc221750d1c" (UID: "499f9e94-a738-484d-ae4b-0cc221750d1c"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:59.167182 master-1 kubenswrapper[4771]: I1011 10:54:59.167103 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499f9e94-a738-484d-ae4b-0cc221750d1c-kube-api-access-k5hg2" (OuterVolumeSpecName: "kube-api-access-k5hg2") pod "499f9e94-a738-484d-ae4b-0cc221750d1c" (UID: "499f9e94-a738-484d-ae4b-0cc221750d1c"). InnerVolumeSpecName "kube-api-access-k5hg2". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:54:59.182593 master-1 kubenswrapper[4771]: I1011 10:54:59.182530 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "499f9e94-a738-484d-ae4b-0cc221750d1c" (UID: "499f9e94-a738-484d-ae4b-0cc221750d1c"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:59.185187 master-1 kubenswrapper[4771]: I1011 10:54:59.185114 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37" (OuterVolumeSpecName: "glance") pod "499f9e94-a738-484d-ae4b-0cc221750d1c" (UID: "499f9e94-a738-484d-ae4b-0cc221750d1c"). InnerVolumeSpecName "pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76". PluginName "kubernetes.io/csi", VolumeGidValue "" Oct 11 10:54:59.211554 master-1 kubenswrapper[4771]: I1011 10:54:59.211431 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-config-data" (OuterVolumeSpecName: "config-data") pod "499f9e94-a738-484d-ae4b-0cc221750d1c" (UID: "499f9e94-a738-484d-ae4b-0cc221750d1c"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:59.225718 master-1 kubenswrapper[4771]: I1011 10:54:59.225616 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-internal-tls-certs" (OuterVolumeSpecName: "internal-tls-certs") pod "499f9e94-a738-484d-ae4b-0cc221750d1c" (UID: "499f9e94-a738-484d-ae4b-0cc221750d1c"). InnerVolumeSpecName "internal-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:54:59.263331 master-1 kubenswrapper[4771]: I1011 10:54:59.263252 4771 reconciler_common.go:286] "operationExecutor.UnmountDevice started for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") on node \"master-1\" " Oct 11 10:54:59.263331 master-1 kubenswrapper[4771]: I1011 10:54:59.263302 4771 reconciler_common.go:293] "Volume detached for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/499f9e94-a738-484d-ae4b-0cc221750d1c-logs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:59.263331 master-1 kubenswrapper[4771]: I1011 10:54:59.263317 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-k5hg2\" (UniqueName: \"kubernetes.io/projected/499f9e94-a738-484d-ae4b-0cc221750d1c-kube-api-access-k5hg2\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:59.263331 master-1 kubenswrapper[4771]: I1011 10:54:59.263330 4771 reconciler_common.go:293] "Volume detached for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-internal-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:59.263331 master-1 kubenswrapper[4771]: I1011 10:54:59.263339 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:59.263331 master-1 kubenswrapper[4771]: I1011 10:54:59.263348 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:59.263331 master-1 kubenswrapper[4771]: I1011 10:54:59.263377 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/499f9e94-a738-484d-ae4b-0cc221750d1c-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:59.281511 master-1 kubenswrapper[4771]: I1011 10:54:59.281438 4771 csi_attacher.go:630] kubernetes.io/csi: attacher.UnmountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping UnmountDevice... Oct 11 10:54:59.282298 master-1 kubenswrapper[4771]: I1011 10:54:59.281660 4771 operation_generator.go:917] UnmountDevice succeeded for volume "pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76" (UniqueName: "kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37") on node "master-1" Oct 11 10:54:59.365317 master-1 kubenswrapper[4771]: I1011 10:54:59.365209 4771 reconciler_common.go:293] "Volume detached for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") on node \"master-1\" DevicePath \"\"" Oct 11 10:54:59.443193 master-1 kubenswrapper[4771]: I1011 10:54:59.442956 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.445210 master-1 kubenswrapper[4771]: I1011 10:54:59.444436 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"499f9e94-a738-484d-ae4b-0cc221750d1c","Type":"ContainerDied","Data":"ee8592262e70401d099ff1b266023cdb236d7c7195e76597576b3cf0944d23f5"} Oct 11 10:54:59.445210 master-1 kubenswrapper[4771]: I1011 10:54:59.444539 4771 scope.go:117] "RemoveContainer" containerID="b479c48e028ed10f47dcf8ff360fd70182a69875e2d6e7028a9c345aed74bb52" Oct 11 10:54:59.445210 master-1 kubenswrapper[4771]: I1011 10:54:59.444708 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 10:54:59.483127 master-1 kubenswrapper[4771]: I1011 10:54:59.482607 4771 scope.go:117] "RemoveContainer" containerID="2d4fd9e07f37d7d0e4c5b7147d47642c209dd291fd8ea33730298efb1acb5aa4" Oct 11 10:54:59.507882 master-1 kubenswrapper[4771]: I1011 10:54:59.507814 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:54:59.513764 master-1 kubenswrapper[4771]: I1011 10:54:59.513723 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:54:59.551549 master-1 kubenswrapper[4771]: I1011 10:54:59.551429 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:54:59.552137 master-1 kubenswrapper[4771]: E1011 10:54:59.552112 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-log" Oct 11 10:54:59.552243 master-1 kubenswrapper[4771]: I1011 10:54:59.552229 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-log" Oct 11 10:54:59.552402 master-1 kubenswrapper[4771]: E1011 10:54:59.552387 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-httpd" Oct 11 10:54:59.552516 master-1 kubenswrapper[4771]: I1011 10:54:59.552498 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-httpd" Oct 11 10:54:59.552836 master-1 kubenswrapper[4771]: I1011 10:54:59.552810 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-httpd" Oct 11 10:54:59.552959 master-1 kubenswrapper[4771]: I1011 10:54:59.552944 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-log" Oct 11 10:54:59.554309 master-1 kubenswrapper[4771]: I1011 10:54:59.554289 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.557676 master-1 kubenswrapper[4771]: I1011 10:54:59.557610 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-glance-default-internal-svc" Oct 11 10:54:59.558702 master-1 kubenswrapper[4771]: I1011 10:54:59.558012 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"glance-b5802-default-internal-config-data" Oct 11 10:54:59.577934 master-1 kubenswrapper[4771]: I1011 10:54:59.577855 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:54:59.679892 master-1 kubenswrapper[4771]: I1011 10:54:59.679821 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.679892 master-1 kubenswrapper[4771]: I1011 10:54:59.679898 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-logs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.680258 master-1 kubenswrapper[4771]: I1011 10:54:59.679927 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2z9\" (UniqueName: \"kubernetes.io/projected/2deabbe8-397d-495c-aef9-afe91b4e9eeb-kube-api-access-2g2z9\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.680258 master-1 kubenswrapper[4771]: I1011 10:54:59.679953 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-scripts\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.680258 master-1 kubenswrapper[4771]: I1011 10:54:59.679980 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-config-data\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.680258 master-1 kubenswrapper[4771]: I1011 10:54:59.680000 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-combined-ca-bundle\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.680258 master-1 kubenswrapper[4771]: I1011 10:54:59.680049 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-httpd-run\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.680258 master-1 kubenswrapper[4771]: I1011 10:54:59.680084 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-internal-tls-certs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.782007 master-1 kubenswrapper[4771]: I1011 10:54:59.781811 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.782007 master-1 kubenswrapper[4771]: I1011 10:54:59.781941 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-logs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.782007 master-1 kubenswrapper[4771]: I1011 10:54:59.781983 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2z9\" (UniqueName: \"kubernetes.io/projected/2deabbe8-397d-495c-aef9-afe91b4e9eeb-kube-api-access-2g2z9\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.782552 master-1 kubenswrapper[4771]: I1011 10:54:59.782021 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-scripts\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.782552 master-1 kubenswrapper[4771]: I1011 10:54:59.782068 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-config-data\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.782552 master-1 kubenswrapper[4771]: I1011 10:54:59.782104 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-combined-ca-bundle\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.782552 master-1 kubenswrapper[4771]: I1011 10:54:59.782203 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-httpd-run\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.782552 master-1 kubenswrapper[4771]: I1011 10:54:59.782319 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-internal-tls-certs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.784424 master-1 kubenswrapper[4771]: I1011 10:54:59.784318 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-httpd-run\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.784875 master-1 kubenswrapper[4771]: I1011 10:54:59.784821 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-logs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.786314 master-1 kubenswrapper[4771]: I1011 10:54:59.786257 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:54:59.786515 master-1 kubenswrapper[4771]: I1011 10:54:59.786326 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/319ddbbf14dc29e9dbd7eec9a997b70a9a11c6eca7f6496495d34ea4ac3ccad0/globalmount\"" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.787315 master-1 kubenswrapper[4771]: I1011 10:54:59.787240 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-internal-tls-certs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.790397 master-1 kubenswrapper[4771]: I1011 10:54:59.789993 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-scripts\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.791748 master-1 kubenswrapper[4771]: I1011 10:54:59.791700 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-config-data\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.804864 master-1 kubenswrapper[4771]: I1011 10:54:59.804622 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-combined-ca-bundle\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:54:59.815024 master-1 kubenswrapper[4771]: I1011 10:54:59.814953 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2z9\" (UniqueName: \"kubernetes.io/projected/2deabbe8-397d-495c-aef9-afe91b4e9eeb-kube-api-access-2g2z9\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:00.455243 master-1 kubenswrapper[4771]: I1011 10:55:00.455107 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" path="/var/lib/kubelet/pods/499f9e94-a738-484d-ae4b-0cc221750d1c/volumes" Oct 11 10:55:01.243960 master-1 kubenswrapper[4771]: I1011 10:55:01.243876 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:01.380933 master-1 kubenswrapper[4771]: I1011 10:55:01.380780 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:02.162111 master-1 kubenswrapper[4771]: I1011 10:55:02.162032 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:55:02.168074 master-1 kubenswrapper[4771]: W1011 10:55:02.167999 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2deabbe8_397d_495c_aef9_afe91b4e9eeb.slice/crio-f7c97ec5b3e0ca2c1b2ecfb01745e5105213a6f53e9a75590979cbcd8d5e7e3f WatchSource:0}: Error finding container f7c97ec5b3e0ca2c1b2ecfb01745e5105213a6f53e9a75590979cbcd8d5e7e3f: Status 404 returned error can't find the container with id f7c97ec5b3e0ca2c1b2ecfb01745e5105213a6f53e9a75590979cbcd8d5e7e3f Oct 11 10:55:02.474727 master-1 kubenswrapper[4771]: I1011 10:55:02.474568 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"2deabbe8-397d-495c-aef9-afe91b4e9eeb","Type":"ContainerStarted","Data":"f7c97ec5b3e0ca2c1b2ecfb01745e5105213a6f53e9a75590979cbcd8d5e7e3f"} Oct 11 10:55:03.485164 master-1 kubenswrapper[4771]: I1011 10:55:03.485084 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"2deabbe8-397d-495c-aef9-afe91b4e9eeb","Type":"ContainerStarted","Data":"20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57"} Oct 11 10:55:03.485164 master-1 kubenswrapper[4771]: I1011 10:55:03.485148 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"2deabbe8-397d-495c-aef9-afe91b4e9eeb","Type":"ContainerStarted","Data":"9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e"} Oct 11 10:55:03.530731 master-1 kubenswrapper[4771]: I1011 10:55:03.530572 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/glance-b5802-default-internal-api-1" podStartSLOduration=4.530539438 podStartE2EDuration="4.530539438s" podCreationTimestamp="2025-10-11 10:54:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:55:03.52298718 +0000 UTC m=+1735.497213631" watchObservedRunningTime="2025-10-11 10:55:03.530539438 +0000 UTC m=+1735.504765909" Oct 11 10:55:04.435784 master-1 kubenswrapper[4771]: I1011 10:55:04.434885 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/neutron-7887b79bcd-stzg5"] Oct 11 10:55:04.438141 master-1 kubenswrapper[4771]: I1011 10:55:04.438075 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.443291 master-1 kubenswrapper[4771]: I1011 10:55:04.442531 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"neutron-httpd-config" Oct 11 10:55:04.443291 master-1 kubenswrapper[4771]: I1011 10:55:04.442826 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-neutron-ovndbs" Oct 11 10:55:04.444279 master-1 kubenswrapper[4771]: I1011 10:55:04.444206 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"neutron-config" Oct 11 10:55:04.466104 master-1 kubenswrapper[4771]: I1011 10:55:04.466057 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/neutron-7887b79bcd-stzg5"] Oct 11 10:55:04.595564 master-1 kubenswrapper[4771]: I1011 10:55:04.595481 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-httpd-config\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.596844 master-1 kubenswrapper[4771]: I1011 10:55:04.595640 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-config\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.597756 master-1 kubenswrapper[4771]: I1011 10:55:04.597724 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-ovndb-tls-certs\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.598105 master-1 kubenswrapper[4771]: I1011 10:55:04.598074 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4zt7\" (UniqueName: \"kubernetes.io/projected/362d815c-c6ec-48b0-9891-85d06ad00aed-kube-api-access-n4zt7\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.598196 master-1 kubenswrapper[4771]: I1011 10:55:04.598168 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-combined-ca-bundle\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.702460 master-1 kubenswrapper[4771]: I1011 10:55:04.702237 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-httpd-config\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.703040 master-1 kubenswrapper[4771]: I1011 10:55:04.702976 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-config\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.703979 master-1 kubenswrapper[4771]: I1011 10:55:04.703926 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-ovndb-tls-certs\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.704125 master-1 kubenswrapper[4771]: I1011 10:55:04.704079 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-n4zt7\" (UniqueName: \"kubernetes.io/projected/362d815c-c6ec-48b0-9891-85d06ad00aed-kube-api-access-n4zt7\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.704345 master-1 kubenswrapper[4771]: I1011 10:55:04.704252 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-combined-ca-bundle\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.708062 master-1 kubenswrapper[4771]: I1011 10:55:04.707190 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-httpd-config\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.708725 master-1 kubenswrapper[4771]: I1011 10:55:04.708669 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-config\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.713604 master-1 kubenswrapper[4771]: I1011 10:55:04.713413 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-ovndb-tls-certs\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.714096 master-1 kubenswrapper[4771]: I1011 10:55:04.714048 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-combined-ca-bundle\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.730519 master-1 kubenswrapper[4771]: I1011 10:55:04.730457 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4zt7\" (UniqueName: \"kubernetes.io/projected/362d815c-c6ec-48b0-9891-85d06ad00aed-kube-api-access-n4zt7\") pod \"neutron-7887b79bcd-stzg5\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.749364 master-1 kubenswrapper[4771]: I1011 10:55:04.749300 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/cinder-b5802-api-0"] Oct 11 10:55:04.751268 master-1 kubenswrapper[4771]: I1011 10:55:04.751247 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/cinder-b5802-api-0" Oct 11 10:55:04.754036 master-1 kubenswrapper[4771]: I1011 10:55:04.753989 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cinder-b5802-config-data" Oct 11 10:55:04.754293 master-1 kubenswrapper[4771]: I1011 10:55:04.754268 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cinder-b5802-api-config-data" Oct 11 10:55:04.754576 master-1 kubenswrapper[4771]: I1011 10:55:04.754552 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cinder-b5802-scripts" Oct 11 10:55:04.764432 master-1 kubenswrapper[4771]: I1011 10:55:04.764078 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/cinder-b5802-api-0"] Oct 11 10:55:04.765824 master-1 kubenswrapper[4771]: I1011 10:55:04.765734 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:04.911333 master-1 kubenswrapper[4771]: I1011 10:55:04.911075 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-machine-id\" (UniqueName: \"kubernetes.io/host-path/478147ef-a0d7-4c37-952c-3fc3a23775db-etc-machine-id\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:04.911333 master-1 kubenswrapper[4771]: I1011 10:55:04.911197 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:04.911333 master-1 kubenswrapper[4771]: I1011 10:55:04.911256 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data-custom\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:04.911333 master-1 kubenswrapper[4771]: I1011 10:55:04.911297 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/478147ef-a0d7-4c37-952c-3fc3a23775db-logs\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:04.911945 master-1 kubenswrapper[4771]: I1011 10:55:04.911867 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvlr\" (UniqueName: \"kubernetes.io/projected/478147ef-a0d7-4c37-952c-3fc3a23775db-kube-api-access-svvlr\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:04.912216 master-1 kubenswrapper[4771]: I1011 10:55:04.911981 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-scripts\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:04.912216 master-1 kubenswrapper[4771]: I1011 10:55:04.912114 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-combined-ca-bundle\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:04.920284 master-1 kubenswrapper[4771]: I1011 10:55:04.920202 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:55:05.013926 master-1 kubenswrapper[4771]: I1011 10:55:05.013741 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.013926 master-1 kubenswrapper[4771]: I1011 10:55:05.013823 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data-custom\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.013926 master-1 kubenswrapper[4771]: I1011 10:55:05.013873 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/478147ef-a0d7-4c37-952c-3fc3a23775db-logs\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.014390 master-1 kubenswrapper[4771]: I1011 10:55:05.014009 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-svvlr\" (UniqueName: \"kubernetes.io/projected/478147ef-a0d7-4c37-952c-3fc3a23775db-kube-api-access-svvlr\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.014390 master-1 kubenswrapper[4771]: I1011 10:55:05.014047 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-scripts\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.014390 master-1 kubenswrapper[4771]: I1011 10:55:05.014105 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-combined-ca-bundle\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.014390 master-1 kubenswrapper[4771]: I1011 10:55:05.014167 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-machine-id\" (UniqueName: \"kubernetes.io/host-path/478147ef-a0d7-4c37-952c-3fc3a23775db-etc-machine-id\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.015248 master-1 kubenswrapper[4771]: I1011 10:55:05.015159 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-machine-id\" (UniqueName: \"kubernetes.io/host-path/478147ef-a0d7-4c37-952c-3fc3a23775db-etc-machine-id\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.015697 master-1 kubenswrapper[4771]: I1011 10:55:05.015637 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/478147ef-a0d7-4c37-952c-3fc3a23775db-logs\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.019767 master-1 kubenswrapper[4771]: I1011 10:55:05.019710 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-scripts\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.019926 master-1 kubenswrapper[4771]: I1011 10:55:05.019869 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data-custom\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.020055 master-1 kubenswrapper[4771]: I1011 10:55:05.019985 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-combined-ca-bundle\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.023090 master-1 kubenswrapper[4771]: I1011 10:55:05.023053 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.053581 master-1 kubenswrapper[4771]: I1011 10:55:05.053537 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvlr\" (UniqueName: \"kubernetes.io/projected/478147ef-a0d7-4c37-952c-3fc3a23775db-kube-api-access-svvlr\") pod \"cinder-b5802-api-0\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.168808 master-1 kubenswrapper[4771]: I1011 10:55:05.168737 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/cinder-b5802-api-0" Oct 11 10:55:05.254390 master-1 kubenswrapper[4771]: I1011 10:55:05.254271 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/placement-6b597cbbf8-8j29d" Oct 11 10:55:05.469967 master-1 kubenswrapper[4771]: W1011 10:55:05.469757 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod362d815c_c6ec_48b0_9891_85d06ad00aed.slice/crio-f93602e6ba46cd0010c3c32ac26a6e16a985dc38b7a78af0515d53b800f6c9e5 WatchSource:0}: Error finding container f93602e6ba46cd0010c3c32ac26a6e16a985dc38b7a78af0515d53b800f6c9e5: Status 404 returned error can't find the container with id f93602e6ba46cd0010c3c32ac26a6e16a985dc38b7a78af0515d53b800f6c9e5 Oct 11 10:55:05.479055 master-1 kubenswrapper[4771]: I1011 10:55:05.478987 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/neutron-7887b79bcd-stzg5"] Oct 11 10:55:05.509555 master-1 kubenswrapper[4771]: I1011 10:55:05.509474 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-7887b79bcd-stzg5" event={"ID":"362d815c-c6ec-48b0-9891-85d06ad00aed","Type":"ContainerStarted","Data":"f93602e6ba46cd0010c3c32ac26a6e16a985dc38b7a78af0515d53b800f6c9e5"} Oct 11 10:55:05.688174 master-1 kubenswrapper[4771]: I1011 10:55:05.683065 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/cinder-b5802-api-0"] Oct 11 10:55:06.525556 master-1 kubenswrapper[4771]: I1011 10:55:06.525371 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/cinder-b5802-api-0" event={"ID":"478147ef-a0d7-4c37-952c-3fc3a23775db","Type":"ContainerStarted","Data":"48979e1f4a2c3ea1ab7c6b56d19ad482a69e0c44021b08cc343a04080547c276"} Oct 11 10:55:06.528536 master-1 kubenswrapper[4771]: I1011 10:55:06.528441 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-7887b79bcd-stzg5" event={"ID":"362d815c-c6ec-48b0-9891-85d06ad00aed","Type":"ContainerStarted","Data":"99502f3eb6699cc67bcf11374ee8446bc01a1a157ce8024301c91ebed596f3f2"} Oct 11 10:55:06.528609 master-1 kubenswrapper[4771]: I1011 10:55:06.528549 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-7887b79bcd-stzg5" event={"ID":"362d815c-c6ec-48b0-9891-85d06ad00aed","Type":"ContainerStarted","Data":"b94dfe1997cbb3d378d19012a9b6401bc1cef35489c7ea7be575908bfe56b3a0"} Oct 11 10:55:06.528696 master-1 kubenswrapper[4771]: I1011 10:55:06.528673 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:06.580813 master-1 kubenswrapper[4771]: I1011 10:55:06.580718 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/neutron-7887b79bcd-stzg5" podStartSLOduration=2.580698106 podStartE2EDuration="2.580698106s" podCreationTimestamp="2025-10-11 10:55:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:55:06.576513685 +0000 UTC m=+1738.550740146" watchObservedRunningTime="2025-10-11 10:55:06.580698106 +0000 UTC m=+1738.554924547" Oct 11 10:55:06.592010 master-1 kubenswrapper[4771]: I1011 10:55:06.591952 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:55:06.592314 master-1 kubenswrapper[4771]: I1011 10:55:06.592268 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/glance-b5802-default-external-api-0" podUID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerName="glance-log" containerID="cri-o://497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b" gracePeriod=30 Oct 11 10:55:06.593148 master-1 kubenswrapper[4771]: I1011 10:55:06.592912 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/glance-b5802-default-external-api-0" podUID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerName="glance-httpd" containerID="cri-o://122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f" gracePeriod=30 Oct 11 10:55:07.539095 master-1 kubenswrapper[4771]: I1011 10:55:07.538938 4771 generic.go:334] "Generic (PLEG): container finished" podID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerID="497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b" exitCode=143 Oct 11 10:55:07.539095 master-1 kubenswrapper[4771]: I1011 10:55:07.539019 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"18861a21-406e-479b-8712-9a62ca2ebf4a","Type":"ContainerDied","Data":"497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b"} Oct 11 10:55:08.597633 master-1 kubenswrapper[4771]: I1011 10:55:08.597552 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/keystone-848fcbb4df-cn592" Oct 11 10:55:10.487866 master-1 kubenswrapper[4771]: I1011 10:55:10.487806 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:10.592887 master-1 kubenswrapper[4771]: I1011 10:55:10.592812 4771 generic.go:334] "Generic (PLEG): container finished" podID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerID="122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f" exitCode=0 Oct 11 10:55:10.592887 master-1 kubenswrapper[4771]: I1011 10:55:10.592884 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"18861a21-406e-479b-8712-9a62ca2ebf4a","Type":"ContainerDied","Data":"122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f"} Oct 11 10:55:10.593328 master-1 kubenswrapper[4771]: I1011 10:55:10.592896 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:10.593328 master-1 kubenswrapper[4771]: I1011 10:55:10.592944 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"18861a21-406e-479b-8712-9a62ca2ebf4a","Type":"ContainerDied","Data":"51aa777863a3d17bf81dc45f1659ccef0c9c30b6b9bf5305b555b52a6a626104"} Oct 11 10:55:10.593328 master-1 kubenswrapper[4771]: I1011 10:55:10.592972 4771 scope.go:117] "RemoveContainer" containerID="122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f" Oct 11 10:55:10.600447 master-1 kubenswrapper[4771]: I1011 10:55:10.600392 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-combined-ca-bundle\") pod \"18861a21-406e-479b-8712-9a62ca2ebf4a\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " Oct 11 10:55:10.600543 master-1 kubenswrapper[4771]: I1011 10:55:10.600531 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jlzm\" (UniqueName: \"kubernetes.io/projected/18861a21-406e-479b-8712-9a62ca2ebf4a-kube-api-access-5jlzm\") pod \"18861a21-406e-479b-8712-9a62ca2ebf4a\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " Oct 11 10:55:10.600640 master-1 kubenswrapper[4771]: I1011 10:55:10.600613 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-httpd-run\") pod \"18861a21-406e-479b-8712-9a62ca2ebf4a\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " Oct 11 10:55:10.600713 master-1 kubenswrapper[4771]: I1011 10:55:10.600657 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-public-tls-certs\") pod \"18861a21-406e-479b-8712-9a62ca2ebf4a\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " Oct 11 10:55:10.600765 master-1 kubenswrapper[4771]: I1011 10:55:10.600717 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-scripts\") pod \"18861a21-406e-479b-8712-9a62ca2ebf4a\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " Oct 11 10:55:10.600809 master-1 kubenswrapper[4771]: I1011 10:55:10.600769 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-config-data\") pod \"18861a21-406e-479b-8712-9a62ca2ebf4a\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " Oct 11 10:55:10.600853 master-1 kubenswrapper[4771]: I1011 10:55:10.600828 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-logs\") pod \"18861a21-406e-479b-8712-9a62ca2ebf4a\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " Oct 11 10:55:10.601094 master-1 kubenswrapper[4771]: I1011 10:55:10.601015 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"glance\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"18861a21-406e-479b-8712-9a62ca2ebf4a\" (UID: \"18861a21-406e-479b-8712-9a62ca2ebf4a\") " Oct 11 10:55:10.602536 master-1 kubenswrapper[4771]: I1011 10:55:10.602498 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-httpd-run" (OuterVolumeSpecName: "httpd-run") pod "18861a21-406e-479b-8712-9a62ca2ebf4a" (UID: "18861a21-406e-479b-8712-9a62ca2ebf4a"). InnerVolumeSpecName "httpd-run". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:10.602647 master-1 kubenswrapper[4771]: I1011 10:55:10.602598 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-logs" (OuterVolumeSpecName: "logs") pod "18861a21-406e-479b-8712-9a62ca2ebf4a" (UID: "18861a21-406e-479b-8712-9a62ca2ebf4a"). InnerVolumeSpecName "logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:10.604982 master-1 kubenswrapper[4771]: I1011 10:55:10.604917 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18861a21-406e-479b-8712-9a62ca2ebf4a-kube-api-access-5jlzm" (OuterVolumeSpecName: "kube-api-access-5jlzm") pod "18861a21-406e-479b-8712-9a62ca2ebf4a" (UID: "18861a21-406e-479b-8712-9a62ca2ebf4a"). InnerVolumeSpecName "kube-api-access-5jlzm". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:10.614200 master-1 kubenswrapper[4771]: I1011 10:55:10.614107 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-scripts" (OuterVolumeSpecName: "scripts") pod "18861a21-406e-479b-8712-9a62ca2ebf4a" (UID: "18861a21-406e-479b-8712-9a62ca2ebf4a"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:10.618109 master-1 kubenswrapper[4771]: I1011 10:55:10.618048 4771 scope.go:117] "RemoveContainer" containerID="497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b" Oct 11 10:55:10.626659 master-1 kubenswrapper[4771]: I1011 10:55:10.626609 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b" (OuterVolumeSpecName: "glance") pod "18861a21-406e-479b-8712-9a62ca2ebf4a" (UID: "18861a21-406e-479b-8712-9a62ca2ebf4a"). InnerVolumeSpecName "pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9". PluginName "kubernetes.io/csi", VolumeGidValue "" Oct 11 10:55:10.628523 master-1 kubenswrapper[4771]: I1011 10:55:10.628473 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "18861a21-406e-479b-8712-9a62ca2ebf4a" (UID: "18861a21-406e-479b-8712-9a62ca2ebf4a"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:10.649300 master-1 kubenswrapper[4771]: I1011 10:55:10.649238 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-config-data" (OuterVolumeSpecName: "config-data") pod "18861a21-406e-479b-8712-9a62ca2ebf4a" (UID: "18861a21-406e-479b-8712-9a62ca2ebf4a"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:10.658248 master-1 kubenswrapper[4771]: I1011 10:55:10.658073 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-public-tls-certs" (OuterVolumeSpecName: "public-tls-certs") pod "18861a21-406e-479b-8712-9a62ca2ebf4a" (UID: "18861a21-406e-479b-8712-9a62ca2ebf4a"). InnerVolumeSpecName "public-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:10.692716 master-1 kubenswrapper[4771]: I1011 10:55:10.692614 4771 scope.go:117] "RemoveContainer" containerID="122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f" Oct 11 10:55:10.693653 master-1 kubenswrapper[4771]: E1011 10:55:10.693575 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f\": container with ID starting with 122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f not found: ID does not exist" containerID="122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f" Oct 11 10:55:10.693892 master-1 kubenswrapper[4771]: I1011 10:55:10.693701 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f"} err="failed to get container status \"122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f\": rpc error: code = NotFound desc = could not find container \"122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f\": container with ID starting with 122528239cb245efc14e6d8ccc44a5f754f4d71a3448c18c3a8d172db64e177f not found: ID does not exist" Oct 11 10:55:10.693892 master-1 kubenswrapper[4771]: I1011 10:55:10.693792 4771 scope.go:117] "RemoveContainer" containerID="497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b" Oct 11 10:55:10.694718 master-1 kubenswrapper[4771]: E1011 10:55:10.694666 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b\": container with ID starting with 497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b not found: ID does not exist" containerID="497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b" Oct 11 10:55:10.694779 master-1 kubenswrapper[4771]: I1011 10:55:10.694719 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b"} err="failed to get container status \"497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b\": rpc error: code = NotFound desc = could not find container \"497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b\": container with ID starting with 497e4e9e0fe8900eab62dbf87a7d7ab2dcbd7bba959b0ab4451a71e7c8f7461b not found: ID does not exist" Oct 11 10:55:10.705039 master-1 kubenswrapper[4771]: I1011 10:55:10.704993 4771 reconciler_common.go:293] "Volume detached for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-public-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:10.705039 master-1 kubenswrapper[4771]: I1011 10:55:10.705038 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:10.705259 master-1 kubenswrapper[4771]: I1011 10:55:10.705221 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:10.705307 master-1 kubenswrapper[4771]: I1011 10:55:10.705241 4771 reconciler_common.go:293] "Volume detached for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-logs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:10.705574 master-1 kubenswrapper[4771]: I1011 10:55:10.705469 4771 reconciler_common.go:286] "operationExecutor.UnmountDevice started for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") on node \"master-1\" " Oct 11 10:55:10.705574 master-1 kubenswrapper[4771]: I1011 10:55:10.705494 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18861a21-406e-479b-8712-9a62ca2ebf4a-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:10.705574 master-1 kubenswrapper[4771]: I1011 10:55:10.705508 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-5jlzm\" (UniqueName: \"kubernetes.io/projected/18861a21-406e-479b-8712-9a62ca2ebf4a-kube-api-access-5jlzm\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:10.705574 master-1 kubenswrapper[4771]: I1011 10:55:10.705520 4771 reconciler_common.go:293] "Volume detached for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/18861a21-406e-479b-8712-9a62ca2ebf4a-httpd-run\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:10.732773 master-1 kubenswrapper[4771]: I1011 10:55:10.732637 4771 csi_attacher.go:630] kubernetes.io/csi: attacher.UnmountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping UnmountDevice... Oct 11 10:55:10.733054 master-1 kubenswrapper[4771]: I1011 10:55:10.733005 4771 operation_generator.go:917] UnmountDevice succeeded for volume "pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9" (UniqueName: "kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b") on node "master-1" Oct 11 10:55:10.807869 master-1 kubenswrapper[4771]: I1011 10:55:10.807811 4771 reconciler_common.go:293] "Volume detached for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:10.847909 master-1 kubenswrapper[4771]: I1011 10:55:10.847825 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/neutron-748bbfcf89-vpkvr"] Oct 11 10:55:10.848313 master-1 kubenswrapper[4771]: E1011 10:55:10.848202 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerName="glance-httpd" Oct 11 10:55:10.848313 master-1 kubenswrapper[4771]: I1011 10:55:10.848220 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerName="glance-httpd" Oct 11 10:55:10.848313 master-1 kubenswrapper[4771]: E1011 10:55:10.848245 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerName="glance-log" Oct 11 10:55:10.848313 master-1 kubenswrapper[4771]: I1011 10:55:10.848255 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerName="glance-log" Oct 11 10:55:10.848526 master-1 kubenswrapper[4771]: I1011 10:55:10.848468 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerName="glance-log" Oct 11 10:55:10.848526 master-1 kubenswrapper[4771]: I1011 10:55:10.848496 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="18861a21-406e-479b-8712-9a62ca2ebf4a" containerName="glance-httpd" Oct 11 10:55:10.849688 master-1 kubenswrapper[4771]: I1011 10:55:10.849650 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:10.852487 master-1 kubenswrapper[4771]: I1011 10:55:10.852411 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-neutron-internal-svc" Oct 11 10:55:10.852487 master-1 kubenswrapper[4771]: I1011 10:55:10.852457 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-neutron-public-svc" Oct 11 10:55:10.864933 master-1 kubenswrapper[4771]: I1011 10:55:10.864878 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/neutron-748bbfcf89-vpkvr"] Oct 11 10:55:10.908841 master-1 kubenswrapper[4771]: I1011 10:55:10.908623 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-httpd-config\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:10.908948 master-1 kubenswrapper[4771]: I1011 10:55:10.908884 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-ovndb-tls-certs\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:10.908997 master-1 kubenswrapper[4771]: I1011 10:55:10.908949 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq6gt\" (UniqueName: \"kubernetes.io/projected/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-kube-api-access-tq6gt\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:10.908997 master-1 kubenswrapper[4771]: I1011 10:55:10.908979 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-combined-ca-bundle\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:10.909069 master-1 kubenswrapper[4771]: I1011 10:55:10.909024 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-internal-tls-certs\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:10.909069 master-1 kubenswrapper[4771]: I1011 10:55:10.909059 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-public-tls-certs\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:10.909155 master-1 kubenswrapper[4771]: I1011 10:55:10.909086 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-config\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:10.938469 master-1 kubenswrapper[4771]: I1011 10:55:10.938404 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:55:10.945455 master-1 kubenswrapper[4771]: I1011 10:55:10.945400 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:55:10.966383 master-1 kubenswrapper[4771]: I1011 10:55:10.966302 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:55:10.967796 master-1 kubenswrapper[4771]: I1011 10:55:10.967760 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:10.973570 master-1 kubenswrapper[4771]: I1011 10:55:10.971281 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-glance-default-public-svc" Oct 11 10:55:10.973570 master-1 kubenswrapper[4771]: I1011 10:55:10.971595 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"glance-b5802-default-external-config-data" Oct 11 10:55:10.991413 master-1 kubenswrapper[4771]: I1011 10:55:10.991341 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:55:11.015439 master-1 kubenswrapper[4771]: I1011 10:55:11.015318 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-internal-tls-certs\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.015657 master-1 kubenswrapper[4771]: I1011 10:55:11.015449 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-public-tls-certs\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.015657 master-1 kubenswrapper[4771]: I1011 10:55:11.015477 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-config\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.015657 master-1 kubenswrapper[4771]: I1011 10:55:11.015528 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-httpd-config\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.015657 master-1 kubenswrapper[4771]: I1011 10:55:11.015590 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-ovndb-tls-certs\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.015657 master-1 kubenswrapper[4771]: I1011 10:55:11.015639 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-tq6gt\" (UniqueName: \"kubernetes.io/projected/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-kube-api-access-tq6gt\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.015657 master-1 kubenswrapper[4771]: I1011 10:55:11.015658 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-combined-ca-bundle\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.021133 master-1 kubenswrapper[4771]: I1011 10:55:11.021066 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-ovndb-tls-certs\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.021772 master-1 kubenswrapper[4771]: I1011 10:55:11.021725 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-public-tls-certs\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.022811 master-1 kubenswrapper[4771]: I1011 10:55:11.022785 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-httpd-config\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.025422 master-1 kubenswrapper[4771]: I1011 10:55:11.023865 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-internal-tls-certs\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.025422 master-1 kubenswrapper[4771]: I1011 10:55:11.025008 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-config\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.026078 master-1 kubenswrapper[4771]: I1011 10:55:11.026045 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-combined-ca-bundle\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.045397 master-1 kubenswrapper[4771]: I1011 10:55:11.043308 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq6gt\" (UniqueName: \"kubernetes.io/projected/5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b-kube-api-access-tq6gt\") pod \"neutron-748bbfcf89-vpkvr\" (UID: \"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b\") " pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.121386 master-1 kubenswrapper[4771]: I1011 10:55:11.120872 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-public-tls-certs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.121386 master-1 kubenswrapper[4771]: I1011 10:55:11.120974 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-config-data\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.121386 master-1 kubenswrapper[4771]: I1011 10:55:11.121054 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-httpd-run\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.121386 master-1 kubenswrapper[4771]: I1011 10:55:11.121138 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-combined-ca-bundle\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.121386 master-1 kubenswrapper[4771]: I1011 10:55:11.121279 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jr4\" (UniqueName: \"kubernetes.io/projected/d3028266-255a-43a3-8bdb-9695ad7cbb30-kube-api-access-85jr4\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.121386 master-1 kubenswrapper[4771]: I1011 10:55:11.121314 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-logs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.121970 master-1 kubenswrapper[4771]: I1011 10:55:11.121654 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-scripts\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.121970 master-1 kubenswrapper[4771]: I1011 10:55:11.121734 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.211420 master-1 kubenswrapper[4771]: I1011 10:55:11.208095 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:11.228386 master-1 kubenswrapper[4771]: I1011 10:55:11.226636 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-scripts\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.228386 master-1 kubenswrapper[4771]: I1011 10:55:11.227364 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.228386 master-1 kubenswrapper[4771]: I1011 10:55:11.227395 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-public-tls-certs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.228386 master-1 kubenswrapper[4771]: I1011 10:55:11.227419 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-config-data\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.228386 master-1 kubenswrapper[4771]: I1011 10:55:11.227461 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-httpd-run\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.228386 master-1 kubenswrapper[4771]: I1011 10:55:11.227505 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-combined-ca-bundle\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.228386 master-1 kubenswrapper[4771]: I1011 10:55:11.227582 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-85jr4\" (UniqueName: \"kubernetes.io/projected/d3028266-255a-43a3-8bdb-9695ad7cbb30-kube-api-access-85jr4\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.228386 master-1 kubenswrapper[4771]: I1011 10:55:11.227603 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-logs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.228386 master-1 kubenswrapper[4771]: I1011 10:55:11.228309 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-logs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.230469 master-1 kubenswrapper[4771]: I1011 10:55:11.230404 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-httpd-run\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.233421 master-1 kubenswrapper[4771]: I1011 10:55:11.231217 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:55:11.233421 master-1 kubenswrapper[4771]: I1011 10:55:11.231287 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/643ba808821ea6db76a2042d255ba68bbc43444ed3cc7e332598424f5540da0c/globalmount\"" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.233421 master-1 kubenswrapper[4771]: I1011 10:55:11.233151 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-scripts\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.238424 master-1 kubenswrapper[4771]: I1011 10:55:11.234844 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-public-tls-certs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.238424 master-1 kubenswrapper[4771]: I1011 10:55:11.238170 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-config-data\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.240255 master-1 kubenswrapper[4771]: I1011 10:55:11.239619 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-combined-ca-bundle\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.251796 master-1 kubenswrapper[4771]: I1011 10:55:11.251676 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jr4\" (UniqueName: \"kubernetes.io/projected/d3028266-255a-43a3-8bdb-9695ad7cbb30-kube-api-access-85jr4\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:11.406077 master-1 kubenswrapper[4771]: I1011 10:55:11.382453 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:11.406077 master-1 kubenswrapper[4771]: I1011 10:55:11.382517 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:11.414771 master-1 kubenswrapper[4771]: I1011 10:55:11.413952 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:11.432102 master-1 kubenswrapper[4771]: I1011 10:55:11.431337 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:11.615000 master-1 kubenswrapper[4771]: I1011 10:55:11.614936 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:11.616204 master-1 kubenswrapper[4771]: I1011 10:55:11.616169 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:11.919407 master-1 kubenswrapper[4771]: I1011 10:55:11.919323 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/neutron-748bbfcf89-vpkvr"] Oct 11 10:55:12.270661 master-1 kubenswrapper[4771]: I1011 10:55:12.270596 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:12.460079 master-1 kubenswrapper[4771]: I1011 10:55:12.458712 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18861a21-406e-479b-8712-9a62ca2ebf4a" path="/var/lib/kubelet/pods/18861a21-406e-479b-8712-9a62ca2ebf4a/volumes" Oct 11 10:55:12.485619 master-1 kubenswrapper[4771]: I1011 10:55:12.485394 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:13.629064 master-1 kubenswrapper[4771]: I1011 10:55:13.628980 4771 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Oct 11 10:55:13.629064 master-1 kubenswrapper[4771]: I1011 10:55:13.629032 4771 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Oct 11 10:55:13.658974 master-1 kubenswrapper[4771]: I1011 10:55:13.657926 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:13.754501 master-1 kubenswrapper[4771]: I1011 10:55:13.754328 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:55:14.935893 master-1 kubenswrapper[4771]: I1011 10:55:14.935814 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ironic-neutron-agent-656ddc8b67-kfkzr"] Oct 11 10:55:14.937849 master-1 kubenswrapper[4771]: I1011 10:55:14.937638 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:14.943914 master-1 kubenswrapper[4771]: I1011 10:55:14.941215 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ironic-ironic-neutron-agent-config-data" Oct 11 10:55:14.985743 master-1 kubenswrapper[4771]: I1011 10:55:14.985328 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ironic-neutron-agent-656ddc8b67-kfkzr"] Oct 11 10:55:15.042075 master-1 kubenswrapper[4771]: I1011 10:55:15.041494 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9gw\" (UniqueName: \"kubernetes.io/projected/c6af8eba-f8bf-47f6-8313-7a902aeb170f-kube-api-access-lx9gw\") pod \"ironic-neutron-agent-656ddc8b67-kfkzr\" (UID: \"c6af8eba-f8bf-47f6-8313-7a902aeb170f\") " pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.042075 master-1 kubenswrapper[4771]: I1011 10:55:15.041800 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6af8eba-f8bf-47f6-8313-7a902aeb170f-config\") pod \"ironic-neutron-agent-656ddc8b67-kfkzr\" (UID: \"c6af8eba-f8bf-47f6-8313-7a902aeb170f\") " pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.042075 master-1 kubenswrapper[4771]: I1011 10:55:15.042001 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6af8eba-f8bf-47f6-8313-7a902aeb170f-combined-ca-bundle\") pod \"ironic-neutron-agent-656ddc8b67-kfkzr\" (UID: \"c6af8eba-f8bf-47f6-8313-7a902aeb170f\") " pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.145201 master-1 kubenswrapper[4771]: I1011 10:55:15.144944 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6af8eba-f8bf-47f6-8313-7a902aeb170f-combined-ca-bundle\") pod \"ironic-neutron-agent-656ddc8b67-kfkzr\" (UID: \"c6af8eba-f8bf-47f6-8313-7a902aeb170f\") " pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.145201 master-1 kubenswrapper[4771]: I1011 10:55:15.145065 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9gw\" (UniqueName: \"kubernetes.io/projected/c6af8eba-f8bf-47f6-8313-7a902aeb170f-kube-api-access-lx9gw\") pod \"ironic-neutron-agent-656ddc8b67-kfkzr\" (UID: \"c6af8eba-f8bf-47f6-8313-7a902aeb170f\") " pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.145201 master-1 kubenswrapper[4771]: I1011 10:55:15.145188 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6af8eba-f8bf-47f6-8313-7a902aeb170f-config\") pod \"ironic-neutron-agent-656ddc8b67-kfkzr\" (UID: \"c6af8eba-f8bf-47f6-8313-7a902aeb170f\") " pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.164003 master-1 kubenswrapper[4771]: I1011 10:55:15.163604 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6af8eba-f8bf-47f6-8313-7a902aeb170f-config\") pod \"ironic-neutron-agent-656ddc8b67-kfkzr\" (UID: \"c6af8eba-f8bf-47f6-8313-7a902aeb170f\") " pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.172309 master-1 kubenswrapper[4771]: I1011 10:55:15.169548 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6af8eba-f8bf-47f6-8313-7a902aeb170f-combined-ca-bundle\") pod \"ironic-neutron-agent-656ddc8b67-kfkzr\" (UID: \"c6af8eba-f8bf-47f6-8313-7a902aeb170f\") " pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.198251 master-1 kubenswrapper[4771]: I1011 10:55:15.198177 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9gw\" (UniqueName: \"kubernetes.io/projected/c6af8eba-f8bf-47f6-8313-7a902aeb170f-kube-api-access-lx9gw\") pod \"ironic-neutron-agent-656ddc8b67-kfkzr\" (UID: \"c6af8eba-f8bf-47f6-8313-7a902aeb170f\") " pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.226912 master-1 kubenswrapper[4771]: I1011 10:55:15.226851 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ironic-7cddc977f5-9ddgm"] Oct 11 10:55:15.230835 master-1 kubenswrapper[4771]: I1011 10:55:15.228892 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.231749 master-1 kubenswrapper[4771]: I1011 10:55:15.231415 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ironic-api-scripts" Oct 11 10:55:15.231749 master-1 kubenswrapper[4771]: I1011 10:55:15.231568 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"rabbitmq-transport-url-ironic-transport" Oct 11 10:55:15.233650 master-1 kubenswrapper[4771]: I1011 10:55:15.233628 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ironic-config-data" Oct 11 10:55:15.234373 master-1 kubenswrapper[4771]: I1011 10:55:15.233879 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ironic-api-config-data" Oct 11 10:55:15.257501 master-1 kubenswrapper[4771]: I1011 10:55:15.249752 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ironic-7cddc977f5-9ddgm"] Oct 11 10:55:15.276759 master-1 kubenswrapper[4771]: I1011 10:55:15.276708 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:15.351434 master-1 kubenswrapper[4771]: I1011 10:55:15.351190 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-custom\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.351434 master-1 kubenswrapper[4771]: I1011 10:55:15.351290 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879970ca-6312-4aec-b8f4-a8a41a0e3797-etc-podinfo\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.351434 master-1 kubenswrapper[4771]: I1011 10:55:15.351378 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.351434 master-1 kubenswrapper[4771]: I1011 10:55:15.351430 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-scripts\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.352031 master-1 kubenswrapper[4771]: I1011 10:55:15.351471 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4pzb\" (UniqueName: \"kubernetes.io/projected/879970ca-6312-4aec-b8f4-a8a41a0e3797-kube-api-access-k4pzb\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.352031 master-1 kubenswrapper[4771]: I1011 10:55:15.351617 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-logs\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.352191 master-1 kubenswrapper[4771]: I1011 10:55:15.352143 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-combined-ca-bundle\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.352397 master-1 kubenswrapper[4771]: I1011 10:55:15.352336 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-merged\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.469469 master-1 kubenswrapper[4771]: I1011 10:55:15.454584 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-logs\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.469469 master-1 kubenswrapper[4771]: I1011 10:55:15.454661 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-combined-ca-bundle\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.469469 master-1 kubenswrapper[4771]: I1011 10:55:15.454730 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-merged\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.469469 master-1 kubenswrapper[4771]: I1011 10:55:15.454785 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-custom\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.469469 master-1 kubenswrapper[4771]: I1011 10:55:15.454814 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879970ca-6312-4aec-b8f4-a8a41a0e3797-etc-podinfo\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.469469 master-1 kubenswrapper[4771]: I1011 10:55:15.454849 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.469469 master-1 kubenswrapper[4771]: I1011 10:55:15.454888 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-scripts\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.469469 master-1 kubenswrapper[4771]: I1011 10:55:15.454919 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-k4pzb\" (UniqueName: \"kubernetes.io/projected/879970ca-6312-4aec-b8f4-a8a41a0e3797-kube-api-access-k4pzb\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.469469 master-1 kubenswrapper[4771]: I1011 10:55:15.455226 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-logs\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.483391 master-1 kubenswrapper[4771]: I1011 10:55:15.475767 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-merged\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.483391 master-1 kubenswrapper[4771]: I1011 10:55:15.476308 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-custom\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.483391 master-1 kubenswrapper[4771]: I1011 10:55:15.480455 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-scripts\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.485425 master-1 kubenswrapper[4771]: I1011 10:55:15.484323 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-combined-ca-bundle\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.503402 master-1 kubenswrapper[4771]: I1011 10:55:15.487508 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4pzb\" (UniqueName: \"kubernetes.io/projected/879970ca-6312-4aec-b8f4-a8a41a0e3797-kube-api-access-k4pzb\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.503402 master-1 kubenswrapper[4771]: I1011 10:55:15.490447 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.508398 master-1 kubenswrapper[4771]: I1011 10:55:15.504097 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879970ca-6312-4aec-b8f4-a8a41a0e3797-etc-podinfo\") pod \"ironic-7cddc977f5-9ddgm\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:15.586694 master-1 kubenswrapper[4771]: I1011 10:55:15.582500 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:16.047318 master-1 kubenswrapper[4771]: I1011 10:55:16.046195 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/heat-cfnapi-b8cb664c5-5zrqf"] Oct 11 10:55:16.047905 master-1 kubenswrapper[4771]: I1011 10:55:16.047837 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.051696 master-1 kubenswrapper[4771]: I1011 10:55:16.051615 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"heat-cfnapi-config-data" Oct 11 10:55:16.052021 master-1 kubenswrapper[4771]: I1011 10:55:16.051957 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"heat-config-data" Oct 11 10:55:16.064331 master-1 kubenswrapper[4771]: I1011 10:55:16.064263 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/heat-cfnapi-b8cb664c5-5zrqf"] Oct 11 10:55:16.084325 master-1 kubenswrapper[4771]: I1011 10:55:16.084234 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:16.085493 master-1 kubenswrapper[4771]: I1011 10:55:16.084867 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="ceilometer-central-agent" containerID="cri-o://1c3861fe4b88a03a7f5a37466aa7554573bbeb56e12a609a086b0d7cf9119e59" gracePeriod=30 Oct 11 10:55:16.085493 master-1 kubenswrapper[4771]: I1011 10:55:16.085043 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="proxy-httpd" containerID="cri-o://3b0917f1e0c562a330c2d467c28336993eadaec7d16c4d43d62cfb2b0ba25b4b" gracePeriod=30 Oct 11 10:55:16.085493 master-1 kubenswrapper[4771]: I1011 10:55:16.085093 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="sg-core" containerID="cri-o://e1413b97fa6430a651152f91df8a84c4a32dbe4f3d81aabb8fb9fea0809e7a16" gracePeriod=30 Oct 11 10:55:16.085493 master-1 kubenswrapper[4771]: I1011 10:55:16.085126 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="ceilometer-notification-agent" containerID="cri-o://0581a657f8cf01879d33e71f4db4cc2df261f4f45ead619016173a151ac38bcc" gracePeriod=30 Oct 11 10:55:16.093943 master-1 kubenswrapper[4771]: I1011 10:55:16.093869 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ceilometer-0" Oct 11 10:55:16.132460 master-1 kubenswrapper[4771]: I1011 10:55:16.131985 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-768f954cfc-gvj8j"] Oct 11 10:55:16.134145 master-1 kubenswrapper[4771]: I1011 10:55:16.134105 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.173157 master-1 kubenswrapper[4771]: I1011 10:55:16.173063 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbk4\" (UniqueName: \"kubernetes.io/projected/1fe7833d-9251-4545-ba68-f58c146188f1-kube-api-access-gzbk4\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.173516 master-1 kubenswrapper[4771]: I1011 10:55:16.173208 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.173516 master-1 kubenswrapper[4771]: I1011 10:55:16.173274 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data-custom\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.173516 master-1 kubenswrapper[4771]: I1011 10:55:16.173323 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-combined-ca-bundle\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.237069 master-1 kubenswrapper[4771]: I1011 10:55:16.222222 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-768f954cfc-gvj8j"] Oct 11 10:55:16.275402 master-1 kubenswrapper[4771]: I1011 10:55:16.275315 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.275800 master-1 kubenswrapper[4771]: I1011 10:55:16.275437 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-nb\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.275800 master-1 kubenswrapper[4771]: I1011 10:55:16.275514 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data-custom\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.275800 master-1 kubenswrapper[4771]: I1011 10:55:16.275544 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-svc\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.275800 master-1 kubenswrapper[4771]: I1011 10:55:16.275568 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-swift-storage-0\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.275800 master-1 kubenswrapper[4771]: I1011 10:55:16.275589 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-combined-ca-bundle\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.275800 master-1 kubenswrapper[4771]: I1011 10:55:16.275645 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zxc\" (UniqueName: \"kubernetes.io/projected/7f2f3d22-d709-4602-bb25-2c17626b75f1-kube-api-access-64zxc\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.275800 master-1 kubenswrapper[4771]: I1011 10:55:16.275694 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbk4\" (UniqueName: \"kubernetes.io/projected/1fe7833d-9251-4545-ba68-f58c146188f1-kube-api-access-gzbk4\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.275800 master-1 kubenswrapper[4771]: I1011 10:55:16.275714 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-config\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.276052 master-1 kubenswrapper[4771]: I1011 10:55:16.275906 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-sb\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.281207 master-1 kubenswrapper[4771]: I1011 10:55:16.281156 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-combined-ca-bundle\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.282097 master-1 kubenswrapper[4771]: I1011 10:55:16.282063 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.287453 master-1 kubenswrapper[4771]: I1011 10:55:16.287311 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data-custom\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.296772 master-1 kubenswrapper[4771]: I1011 10:55:16.296712 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbk4\" (UniqueName: \"kubernetes.io/projected/1fe7833d-9251-4545-ba68-f58c146188f1-kube-api-access-gzbk4\") pod \"heat-cfnapi-b8cb664c5-5zrqf\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.372420 master-1 kubenswrapper[4771]: I1011 10:55:16.372181 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:16.379019 master-1 kubenswrapper[4771]: I1011 10:55:16.378941 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-nb\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.379136 master-1 kubenswrapper[4771]: I1011 10:55:16.379032 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-svc\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.379136 master-1 kubenswrapper[4771]: I1011 10:55:16.379077 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-swift-storage-0\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.379210 master-1 kubenswrapper[4771]: I1011 10:55:16.379156 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-64zxc\" (UniqueName: \"kubernetes.io/projected/7f2f3d22-d709-4602-bb25-2c17626b75f1-kube-api-access-64zxc\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.379342 master-1 kubenswrapper[4771]: I1011 10:55:16.379220 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-config\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.379342 master-1 kubenswrapper[4771]: I1011 10:55:16.379277 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-sb\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.381006 master-1 kubenswrapper[4771]: I1011 10:55:16.380543 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-sb\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.381006 master-1 kubenswrapper[4771]: I1011 10:55:16.380858 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-swift-storage-0\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.382037 master-1 kubenswrapper[4771]: I1011 10:55:16.381959 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-svc\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.382261 master-1 kubenswrapper[4771]: I1011 10:55:16.382163 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-config\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.382887 master-1 kubenswrapper[4771]: I1011 10:55:16.382840 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-nb\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.415079 master-1 kubenswrapper[4771]: I1011 10:55:16.404910 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zxc\" (UniqueName: \"kubernetes.io/projected/7f2f3d22-d709-4602-bb25-2c17626b75f1-kube-api-access-64zxc\") pod \"dnsmasq-dns-768f954cfc-gvj8j\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.459602 master-1 kubenswrapper[4771]: I1011 10:55:16.459320 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:16.670506 master-1 kubenswrapper[4771]: I1011 10:55:16.670396 4771 generic.go:334] "Generic (PLEG): container finished" podID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerID="3b0917f1e0c562a330c2d467c28336993eadaec7d16c4d43d62cfb2b0ba25b4b" exitCode=0 Oct 11 10:55:16.670506 master-1 kubenswrapper[4771]: I1011 10:55:16.670437 4771 generic.go:334] "Generic (PLEG): container finished" podID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerID="e1413b97fa6430a651152f91df8a84c4a32dbe4f3d81aabb8fb9fea0809e7a16" exitCode=2 Oct 11 10:55:16.670506 master-1 kubenswrapper[4771]: I1011 10:55:16.670471 4771 generic.go:334] "Generic (PLEG): container finished" podID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerID="1c3861fe4b88a03a7f5a37466aa7554573bbeb56e12a609a086b0d7cf9119e59" exitCode=0 Oct 11 10:55:16.670506 master-1 kubenswrapper[4771]: I1011 10:55:16.670499 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerDied","Data":"3b0917f1e0c562a330c2d467c28336993eadaec7d16c4d43d62cfb2b0ba25b4b"} Oct 11 10:55:16.671240 master-1 kubenswrapper[4771]: I1011 10:55:16.670531 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerDied","Data":"e1413b97fa6430a651152f91df8a84c4a32dbe4f3d81aabb8fb9fea0809e7a16"} Oct 11 10:55:16.671240 master-1 kubenswrapper[4771]: I1011 10:55:16.670543 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerDied","Data":"1c3861fe4b88a03a7f5a37466aa7554573bbeb56e12a609a086b0d7cf9119e59"} Oct 11 10:55:17.681851 master-1 kubenswrapper[4771]: I1011 10:55:17.681660 4771 generic.go:334] "Generic (PLEG): container finished" podID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerID="0581a657f8cf01879d33e71f4db4cc2df261f4f45ead619016173a151ac38bcc" exitCode=0 Oct 11 10:55:17.681851 master-1 kubenswrapper[4771]: I1011 10:55:17.681724 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerDied","Data":"0581a657f8cf01879d33e71f4db4cc2df261f4f45ead619016173a151ac38bcc"} Oct 11 10:55:22.758342 master-1 kubenswrapper[4771]: I1011 10:55:22.758276 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-748bbfcf89-vpkvr" event={"ID":"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b","Type":"ContainerStarted","Data":"af9ff314458369d3f8145591e9f896c9e3bb8b7fa9ff1992f73b7aee94d96f60"} Oct 11 10:55:22.988217 master-1 kubenswrapper[4771]: I1011 10:55:22.988178 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:23.042135 master-1 kubenswrapper[4771]: I1011 10:55:23.042062 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-sg-core-conf-yaml\") pod \"49ec9c51-e085-4cfa-8ce7-387a02f23731\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " Oct 11 10:55:23.042365 master-1 kubenswrapper[4771]: I1011 10:55:23.042225 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-combined-ca-bundle\") pod \"49ec9c51-e085-4cfa-8ce7-387a02f23731\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " Oct 11 10:55:23.042732 master-1 kubenswrapper[4771]: I1011 10:55:23.042642 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj45w\" (UniqueName: \"kubernetes.io/projected/49ec9c51-e085-4cfa-8ce7-387a02f23731-kube-api-access-rj45w\") pod \"49ec9c51-e085-4cfa-8ce7-387a02f23731\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " Oct 11 10:55:23.042831 master-1 kubenswrapper[4771]: I1011 10:55:23.042750 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-run-httpd\") pod \"49ec9c51-e085-4cfa-8ce7-387a02f23731\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " Oct 11 10:55:23.042831 master-1 kubenswrapper[4771]: I1011 10:55:23.042778 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-config-data\") pod \"49ec9c51-e085-4cfa-8ce7-387a02f23731\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " Oct 11 10:55:23.042831 master-1 kubenswrapper[4771]: I1011 10:55:23.042807 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-scripts\") pod \"49ec9c51-e085-4cfa-8ce7-387a02f23731\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " Oct 11 10:55:23.043022 master-1 kubenswrapper[4771]: I1011 10:55:23.042853 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-log-httpd\") pod \"49ec9c51-e085-4cfa-8ce7-387a02f23731\" (UID: \"49ec9c51-e085-4cfa-8ce7-387a02f23731\") " Oct 11 10:55:23.044191 master-1 kubenswrapper[4771]: I1011 10:55:23.044109 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "49ec9c51-e085-4cfa-8ce7-387a02f23731" (UID: "49ec9c51-e085-4cfa-8ce7-387a02f23731"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:23.045986 master-1 kubenswrapper[4771]: I1011 10:55:23.045925 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "49ec9c51-e085-4cfa-8ce7-387a02f23731" (UID: "49ec9c51-e085-4cfa-8ce7-387a02f23731"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:23.047410 master-1 kubenswrapper[4771]: I1011 10:55:23.047381 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ec9c51-e085-4cfa-8ce7-387a02f23731-kube-api-access-rj45w" (OuterVolumeSpecName: "kube-api-access-rj45w") pod "49ec9c51-e085-4cfa-8ce7-387a02f23731" (UID: "49ec9c51-e085-4cfa-8ce7-387a02f23731"). InnerVolumeSpecName "kube-api-access-rj45w". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:23.047938 master-1 kubenswrapper[4771]: I1011 10:55:23.047910 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-scripts" (OuterVolumeSpecName: "scripts") pod "49ec9c51-e085-4cfa-8ce7-387a02f23731" (UID: "49ec9c51-e085-4cfa-8ce7-387a02f23731"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:23.076733 master-1 kubenswrapper[4771]: I1011 10:55:23.076661 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "49ec9c51-e085-4cfa-8ce7-387a02f23731" (UID: "49ec9c51-e085-4cfa-8ce7-387a02f23731"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:23.118322 master-1 kubenswrapper[4771]: I1011 10:55:23.118230 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "49ec9c51-e085-4cfa-8ce7-387a02f23731" (UID: "49ec9c51-e085-4cfa-8ce7-387a02f23731"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:23.136234 master-1 kubenswrapper[4771]: I1011 10:55:23.136175 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-config-data" (OuterVolumeSpecName: "config-data") pod "49ec9c51-e085-4cfa-8ce7-387a02f23731" (UID: "49ec9c51-e085-4cfa-8ce7-387a02f23731"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:23.146143 master-1 kubenswrapper[4771]: I1011 10:55:23.146072 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:23.147098 master-1 kubenswrapper[4771]: I1011 10:55:23.147069 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:23.147098 master-1 kubenswrapper[4771]: I1011 10:55:23.147092 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-rj45w\" (UniqueName: \"kubernetes.io/projected/49ec9c51-e085-4cfa-8ce7-387a02f23731-kube-api-access-rj45w\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:23.147213 master-1 kubenswrapper[4771]: I1011 10:55:23.147107 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:23.147213 master-1 kubenswrapper[4771]: I1011 10:55:23.147118 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:23.147213 master-1 kubenswrapper[4771]: I1011 10:55:23.147128 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/49ec9c51-e085-4cfa-8ce7-387a02f23731-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:23.147213 master-1 kubenswrapper[4771]: I1011 10:55:23.147137 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/49ec9c51-e085-4cfa-8ce7-387a02f23731-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:23.380976 master-1 kubenswrapper[4771]: I1011 10:55:23.378590 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-768f954cfc-gvj8j"] Oct 11 10:55:23.498739 master-1 kubenswrapper[4771]: I1011 10:55:23.498703 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/heat-cfnapi-b8cb664c5-5zrqf"] Oct 11 10:55:23.505951 master-1 kubenswrapper[4771]: W1011 10:55:23.505661 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e WatchSource:0}: Error finding container fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e: Status 404 returned error can't find the container with id fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e Oct 11 10:55:23.538405 master-1 kubenswrapper[4771]: I1011 10:55:23.538363 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ironic-7cddc977f5-9ddgm"] Oct 11 10:55:23.657599 master-1 kubenswrapper[4771]: W1011 10:55:23.657343 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod879970ca_6312_4aec_b8f4_a8a41a0e3797.slice/crio-3182b36cd0161342e4b24feff5f2f372022de500b18e279f9cd6a7f20bca373c WatchSource:0}: Error finding container 3182b36cd0161342e4b24feff5f2f372022de500b18e279f9cd6a7f20bca373c: Status 404 returned error can't find the container with id 3182b36cd0161342e4b24feff5f2f372022de500b18e279f9cd6a7f20bca373c Oct 11 10:55:23.679308 master-1 kubenswrapper[4771]: I1011 10:55:23.679253 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ironic-neutron-agent-656ddc8b67-kfkzr"] Oct 11 10:55:23.777660 master-1 kubenswrapper[4771]: I1011 10:55:23.777581 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" event={"ID":"7f2f3d22-d709-4602-bb25-2c17626b75f1","Type":"ContainerStarted","Data":"44d3f4f2f792c76343fbbdc36dc9dcbfc23552c6a12758c872ba3f43c891c162"} Oct 11 10:55:23.783082 master-1 kubenswrapper[4771]: I1011 10:55:23.782997 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" event={"ID":"1fe7833d-9251-4545-ba68-f58c146188f1","Type":"ContainerStarted","Data":"fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e"} Oct 11 10:55:23.787668 master-1 kubenswrapper[4771]: I1011 10:55:23.786515 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-748bbfcf89-vpkvr" event={"ID":"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b","Type":"ContainerStarted","Data":"8ea70b801c6579249e824c5ec9cdefec815296a35486c33c1b92b000fae180f4"} Oct 11 10:55:23.787668 master-1 kubenswrapper[4771]: I1011 10:55:23.786568 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-748bbfcf89-vpkvr" event={"ID":"5c7a3a13-f5fd-4cb5-8830-d74a07e1b09b","Type":"ContainerStarted","Data":"d45639345dc639621f85b654973079ba0284184faa5f92ee6c18cf3c74c93a74"} Oct 11 10:55:23.787668 master-1 kubenswrapper[4771]: I1011 10:55:23.786710 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:23.789683 master-1 kubenswrapper[4771]: I1011 10:55:23.789652 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-7cddc977f5-9ddgm" event={"ID":"879970ca-6312-4aec-b8f4-a8a41a0e3797","Type":"ContainerStarted","Data":"3182b36cd0161342e4b24feff5f2f372022de500b18e279f9cd6a7f20bca373c"} Oct 11 10:55:23.792390 master-1 kubenswrapper[4771]: I1011 10:55:23.792324 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/cinder-b5802-api-0" event={"ID":"478147ef-a0d7-4c37-952c-3fc3a23775db","Type":"ContainerStarted","Data":"fff01313b302342cb30f2201b57bb76a5615b3d6076b484b8fb9b7d061e529af"} Oct 11 10:55:23.796280 master-1 kubenswrapper[4771]: I1011 10:55:23.796103 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"49ec9c51-e085-4cfa-8ce7-387a02f23731","Type":"ContainerDied","Data":"ada9da27885a5797d5d7044aa85caf6f5ca08e4448276ab5e54d26349906f001"} Oct 11 10:55:23.796280 master-1 kubenswrapper[4771]: I1011 10:55:23.796168 4771 scope.go:117] "RemoveContainer" containerID="3b0917f1e0c562a330c2d467c28336993eadaec7d16c4d43d62cfb2b0ba25b4b" Oct 11 10:55:23.796280 master-1 kubenswrapper[4771]: I1011 10:55:23.796223 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:23.854909 master-1 kubenswrapper[4771]: I1011 10:55:23.854853 4771 scope.go:117] "RemoveContainer" containerID="e1413b97fa6430a651152f91df8a84c4a32dbe4f3d81aabb8fb9fea0809e7a16" Oct 11 10:55:23.875869 master-1 kubenswrapper[4771]: I1011 10:55:23.875826 4771 scope.go:117] "RemoveContainer" containerID="0581a657f8cf01879d33e71f4db4cc2df261f4f45ead619016173a151ac38bcc" Oct 11 10:55:23.904535 master-1 kubenswrapper[4771]: I1011 10:55:23.904410 4771 scope.go:117] "RemoveContainer" containerID="1c3861fe4b88a03a7f5a37466aa7554573bbeb56e12a609a086b0d7cf9119e59" Oct 11 10:55:23.913222 master-1 kubenswrapper[4771]: W1011 10:55:23.913170 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6af8eba_f8bf_47f6_8313_7a902aeb170f.slice/crio-4e0c21fb2f4105429136935982c5a615f430c2fe86d42023dbe3fb84ea97d400 WatchSource:0}: Error finding container 4e0c21fb2f4105429136935982c5a615f430c2fe86d42023dbe3fb84ea97d400: Status 404 returned error can't find the container with id 4e0c21fb2f4105429136935982c5a615f430c2fe86d42023dbe3fb84ea97d400 Oct 11 10:55:23.970318 master-1 kubenswrapper[4771]: I1011 10:55:23.970220 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:23.984617 master-1 kubenswrapper[4771]: I1011 10:55:23.984189 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:23.993936 master-1 kubenswrapper[4771]: I1011 10:55:23.993863 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.004527 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: E1011 10:55:24.005008 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="ceilometer-notification-agent" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.005025 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="ceilometer-notification-agent" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: E1011 10:55:24.005053 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="ceilometer-central-agent" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.005062 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="ceilometer-central-agent" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: E1011 10:55:24.005087 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="sg-core" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.005096 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="sg-core" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: E1011 10:55:24.005117 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="proxy-httpd" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.005126 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="proxy-httpd" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.005317 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="ceilometer-notification-agent" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.005343 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="ceilometer-central-agent" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.005378 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="proxy-httpd" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.005392 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" containerName="sg-core" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.005624 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/neutron-748bbfcf89-vpkvr" podStartSLOduration=14.005596828 podStartE2EDuration="14.005596828s" podCreationTimestamp="2025-10-11 10:55:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:55:23.992892298 +0000 UTC m=+1755.967118749" watchObservedRunningTime="2025-10-11 10:55:24.005596828 +0000 UTC m=+1755.979823269" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.007320 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.011160 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 10:55:24.012489 master-1 kubenswrapper[4771]: I1011 10:55:24.011574 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 10:55:24.036400 master-1 kubenswrapper[4771]: I1011 10:55:24.031231 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:24.069399 master-1 kubenswrapper[4771]: I1011 10:55:24.065765 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-run-httpd\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.069399 master-1 kubenswrapper[4771]: I1011 10:55:24.065869 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.069399 master-1 kubenswrapper[4771]: I1011 10:55:24.065914 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-log-httpd\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.069399 master-1 kubenswrapper[4771]: I1011 10:55:24.065973 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7f7t\" (UniqueName: \"kubernetes.io/projected/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-kube-api-access-v7f7t\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.069399 master-1 kubenswrapper[4771]: I1011 10:55:24.066022 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-scripts\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.069399 master-1 kubenswrapper[4771]: I1011 10:55:24.066053 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-config-data\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.069399 master-1 kubenswrapper[4771]: I1011 10:55:24.066077 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.170692 master-1 kubenswrapper[4771]: I1011 10:55:24.170401 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-run-httpd\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.171122 master-1 kubenswrapper[4771]: I1011 10:55:24.171072 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.171295 master-1 kubenswrapper[4771]: I1011 10:55:24.171259 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-log-httpd\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.172371 master-1 kubenswrapper[4771]: I1011 10:55:24.172048 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-run-httpd\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.172371 master-1 kubenswrapper[4771]: I1011 10:55:24.172164 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-log-httpd\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.172727 master-1 kubenswrapper[4771]: I1011 10:55:24.172665 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-v7f7t\" (UniqueName: \"kubernetes.io/projected/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-kube-api-access-v7f7t\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.172969 master-1 kubenswrapper[4771]: I1011 10:55:24.172868 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-scripts\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.183277 master-1 kubenswrapper[4771]: I1011 10:55:24.172945 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-config-data\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.183277 master-1 kubenswrapper[4771]: I1011 10:55:24.173092 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.183277 master-1 kubenswrapper[4771]: I1011 10:55:24.181125 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-config-data\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.183277 master-1 kubenswrapper[4771]: I1011 10:55:24.183173 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.204168 master-1 kubenswrapper[4771]: I1011 10:55:24.204094 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-scripts\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.205453 master-1 kubenswrapper[4771]: I1011 10:55:24.205391 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.214399 master-1 kubenswrapper[4771]: I1011 10:55:24.211952 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7f7t\" (UniqueName: \"kubernetes.io/projected/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-kube-api-access-v7f7t\") pod \"ceilometer-0\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " pod="openstack/ceilometer-0" Oct 11 10:55:24.216435 master-1 kubenswrapper[4771]: I1011 10:55:24.216400 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:24.456957 master-1 kubenswrapper[4771]: I1011 10:55:24.456690 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ec9c51-e085-4cfa-8ce7-387a02f23731" path="/var/lib/kubelet/pods/49ec9c51-e085-4cfa-8ce7-387a02f23731/volumes" Oct 11 10:55:24.722081 master-1 kubenswrapper[4771]: I1011 10:55:24.722028 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:24.824433 master-1 kubenswrapper[4771]: I1011 10:55:24.823036 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/cinder-b5802-api-0" event={"ID":"478147ef-a0d7-4c37-952c-3fc3a23775db","Type":"ContainerStarted","Data":"cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80"} Oct 11 10:55:24.824433 master-1 kubenswrapper[4771]: I1011 10:55:24.824328 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/cinder-b5802-api-0" Oct 11 10:55:24.827791 master-1 kubenswrapper[4771]: I1011 10:55:24.827742 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerStarted","Data":"5c043b242e5c8ff65d3ea42d92bba671ec7f6446a265531a8e4be33feddbe4fa"} Oct 11 10:55:24.829901 master-1 kubenswrapper[4771]: I1011 10:55:24.829859 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"d3028266-255a-43a3-8bdb-9695ad7cbb30","Type":"ContainerStarted","Data":"7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474"} Oct 11 10:55:24.829985 master-1 kubenswrapper[4771]: I1011 10:55:24.829902 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"d3028266-255a-43a3-8bdb-9695ad7cbb30","Type":"ContainerStarted","Data":"a58cd7c5986caf2dc23489be67139cc61da2a0165758a7758f40955bc3fa5183"} Oct 11 10:55:24.842696 master-1 kubenswrapper[4771]: I1011 10:55:24.842631 4771 generic.go:334] "Generic (PLEG): container finished" podID="7f2f3d22-d709-4602-bb25-2c17626b75f1" containerID="4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1" exitCode=0 Oct 11 10:55:24.843070 master-1 kubenswrapper[4771]: I1011 10:55:24.843042 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" event={"ID":"7f2f3d22-d709-4602-bb25-2c17626b75f1","Type":"ContainerDied","Data":"4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1"} Oct 11 10:55:24.847041 master-1 kubenswrapper[4771]: I1011 10:55:24.846985 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" event={"ID":"c6af8eba-f8bf-47f6-8313-7a902aeb170f","Type":"ContainerStarted","Data":"4e0c21fb2f4105429136935982c5a615f430c2fe86d42023dbe3fb84ea97d400"} Oct 11 10:55:24.856439 master-1 kubenswrapper[4771]: I1011 10:55:24.856261 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/cinder-b5802-api-0" podStartSLOduration=3.838802839 podStartE2EDuration="20.856232825s" podCreationTimestamp="2025-10-11 10:55:04 +0000 UTC" firstStartedPulling="2025-10-11 10:55:05.692614701 +0000 UTC m=+1737.666841192" lastFinishedPulling="2025-10-11 10:55:22.710044727 +0000 UTC m=+1754.684271178" observedRunningTime="2025-10-11 10:55:24.847017967 +0000 UTC m=+1756.821244418" watchObservedRunningTime="2025-10-11 10:55:24.856232825 +0000 UTC m=+1756.830459266" Oct 11 10:55:25.408720 master-1 kubenswrapper[4771]: I1011 10:55:25.408641 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/heat-api-d647f9c47-x7xc2"] Oct 11 10:55:25.410218 master-1 kubenswrapper[4771]: I1011 10:55:25.410197 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.415102 master-1 kubenswrapper[4771]: I1011 10:55:25.415047 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"heat-api-config-data" Oct 11 10:55:25.436265 master-1 kubenswrapper[4771]: I1011 10:55:25.434983 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/heat-api-d647f9c47-x7xc2"] Oct 11 10:55:25.536377 master-1 kubenswrapper[4771]: I1011 10:55:25.533408 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data-custom\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.536377 master-1 kubenswrapper[4771]: I1011 10:55:25.533635 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-combined-ca-bundle\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.536377 master-1 kubenswrapper[4771]: I1011 10:55:25.533784 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z885b\" (UniqueName: \"kubernetes.io/projected/e0657ee5-2e60-4a96-905e-814f46a72970-kube-api-access-z885b\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.536377 master-1 kubenswrapper[4771]: I1011 10:55:25.533847 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.605832 master-1 kubenswrapper[4771]: I1011 10:55:25.605735 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:25.636122 master-1 kubenswrapper[4771]: I1011 10:55:25.635991 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.636122 master-1 kubenswrapper[4771]: I1011 10:55:25.636085 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data-custom\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.636122 master-1 kubenswrapper[4771]: I1011 10:55:25.636120 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-combined-ca-bundle\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.636638 master-1 kubenswrapper[4771]: I1011 10:55:25.636181 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-z885b\" (UniqueName: \"kubernetes.io/projected/e0657ee5-2e60-4a96-905e-814f46a72970-kube-api-access-z885b\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.645104 master-1 kubenswrapper[4771]: I1011 10:55:25.644034 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.653385 master-1 kubenswrapper[4771]: I1011 10:55:25.648798 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data-custom\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.660759 master-1 kubenswrapper[4771]: I1011 10:55:25.656630 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-z885b\" (UniqueName: \"kubernetes.io/projected/e0657ee5-2e60-4a96-905e-814f46a72970-kube-api-access-z885b\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.665062 master-1 kubenswrapper[4771]: I1011 10:55:25.664996 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-combined-ca-bundle\") pod \"heat-api-d647f9c47-x7xc2\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.731500 master-1 kubenswrapper[4771]: I1011 10:55:25.729459 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:25.871837 master-1 kubenswrapper[4771]: I1011 10:55:25.871721 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerStarted","Data":"33e1159e64df7103066e5f7850051b2adc3d09e823478d0dc1137ddef2aee326"} Oct 11 10:55:25.882039 master-1 kubenswrapper[4771]: I1011 10:55:25.881936 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"d3028266-255a-43a3-8bdb-9695ad7cbb30","Type":"ContainerStarted","Data":"9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c"} Oct 11 10:55:25.892750 master-1 kubenswrapper[4771]: I1011 10:55:25.892647 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" event={"ID":"7f2f3d22-d709-4602-bb25-2c17626b75f1","Type":"ContainerStarted","Data":"f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973"} Oct 11 10:55:25.897042 master-1 kubenswrapper[4771]: I1011 10:55:25.896470 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:25.917511 master-1 kubenswrapper[4771]: I1011 10:55:25.917417 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/glance-b5802-default-external-api-0" podStartSLOduration=15.917388042 podStartE2EDuration="15.917388042s" podCreationTimestamp="2025-10-11 10:55:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:55:25.915550978 +0000 UTC m=+1757.889777429" watchObservedRunningTime="2025-10-11 10:55:25.917388042 +0000 UTC m=+1757.891614483" Oct 11 10:55:26.253611 master-1 kubenswrapper[4771]: I1011 10:55:26.253484 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" podStartSLOduration=10.253453853 podStartE2EDuration="10.253453853s" podCreationTimestamp="2025-10-11 10:55:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:55:25.966784122 +0000 UTC m=+1757.941010563" watchObservedRunningTime="2025-10-11 10:55:26.253453853 +0000 UTC m=+1758.227680294" Oct 11 10:55:26.261004 master-1 kubenswrapper[4771]: I1011 10:55:26.260943 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/heat-api-d647f9c47-x7xc2"] Oct 11 10:55:26.913343 master-1 kubenswrapper[4771]: I1011 10:55:26.913210 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerStarted","Data":"1d0d93b3fc6393dcdc851e8c3921d7c5d5a44cf9e99d331f9e66f61b3c48f59d"} Oct 11 10:55:26.916619 master-1 kubenswrapper[4771]: I1011 10:55:26.914977 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-api-d647f9c47-x7xc2" event={"ID":"e0657ee5-2e60-4a96-905e-814f46a72970","Type":"ContainerStarted","Data":"4d5653f9de27a6b9172b5e020cb9d04e796130d0328e34ab12c5dd9a66c1452e"} Oct 11 10:55:27.123204 master-1 kubenswrapper[4771]: I1011 10:55:27.123114 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/heat-cfnapi-b8cb664c5-5zrqf"] Oct 11 10:55:27.162447 master-1 kubenswrapper[4771]: I1011 10:55:27.162386 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/heat-cfnapi-64fcdf7d54-8r455"] Oct 11 10:55:27.164458 master-1 kubenswrapper[4771]: I1011 10:55:27.164435 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.167718 master-1 kubenswrapper[4771]: I1011 10:55:27.167651 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-heat-cfnapi-internal-svc" Oct 11 10:55:27.168024 master-1 kubenswrapper[4771]: I1011 10:55:27.167996 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-heat-cfnapi-public-svc" Oct 11 10:55:27.186345 master-1 kubenswrapper[4771]: I1011 10:55:27.186286 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-internal-tls-certs\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.186865 master-1 kubenswrapper[4771]: I1011 10:55:27.186688 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-combined-ca-bundle\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.186865 master-1 kubenswrapper[4771]: I1011 10:55:27.186846 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-config-data\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.187061 master-1 kubenswrapper[4771]: I1011 10:55:27.186982 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-config-data-custom\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.187061 master-1 kubenswrapper[4771]: I1011 10:55:27.187042 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-public-tls-certs\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.187142 master-1 kubenswrapper[4771]: I1011 10:55:27.187067 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwt9\" (UniqueName: \"kubernetes.io/projected/2dc94855-37f8-4fa8-a3e1-72808b37f966-kube-api-access-slwt9\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.187180 master-1 kubenswrapper[4771]: I1011 10:55:27.187123 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/heat-cfnapi-64fcdf7d54-8r455"] Oct 11 10:55:27.289503 master-1 kubenswrapper[4771]: I1011 10:55:27.289430 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-combined-ca-bundle\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.289654 master-1 kubenswrapper[4771]: I1011 10:55:27.289580 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-config-data\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.289699 master-1 kubenswrapper[4771]: I1011 10:55:27.289660 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-config-data-custom\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.289743 master-1 kubenswrapper[4771]: I1011 10:55:27.289715 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-public-tls-certs\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.289743 master-1 kubenswrapper[4771]: I1011 10:55:27.289737 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-slwt9\" (UniqueName: \"kubernetes.io/projected/2dc94855-37f8-4fa8-a3e1-72808b37f966-kube-api-access-slwt9\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.289806 master-1 kubenswrapper[4771]: I1011 10:55:27.289797 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-internal-tls-certs\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.294052 master-1 kubenswrapper[4771]: I1011 10:55:27.293919 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-internal-tls-certs\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.308116 master-1 kubenswrapper[4771]: I1011 10:55:27.308067 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-config-data\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.322386 master-1 kubenswrapper[4771]: I1011 10:55:27.318005 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-combined-ca-bundle\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.322386 master-1 kubenswrapper[4771]: I1011 10:55:27.319454 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-public-tls-certs\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.323718 master-1 kubenswrapper[4771]: I1011 10:55:27.323634 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwt9\" (UniqueName: \"kubernetes.io/projected/2dc94855-37f8-4fa8-a3e1-72808b37f966-kube-api-access-slwt9\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.325436 master-1 kubenswrapper[4771]: I1011 10:55:27.325065 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/2dc94855-37f8-4fa8-a3e1-72808b37f966-config-data-custom\") pod \"heat-cfnapi-64fcdf7d54-8r455\" (UID: \"2dc94855-37f8-4fa8-a3e1-72808b37f966\") " pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:27.504626 master-1 kubenswrapper[4771]: I1011 10:55:27.503385 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:28.768709 master-1 kubenswrapper[4771]: I1011 10:55:28.768480 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/glance-b5802-default-internal-api-1" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-log" probeResult="failure" output="Get \"https://10.129.0.126:9292/healthcheck\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 11 10:55:28.768709 master-1 kubenswrapper[4771]: I1011 10:55:28.768523 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/glance-b5802-default-internal-api-1" podUID="499f9e94-a738-484d-ae4b-0cc221750d1c" containerName="glance-httpd" probeResult="failure" output="Get \"https://10.129.0.126:9292/healthcheck\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 11 10:55:28.967234 master-1 kubenswrapper[4771]: I1011 10:55:28.965621 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-operators-8vzsw"] Oct 11 10:55:28.967962 master-1 kubenswrapper[4771]: I1011 10:55:28.967930 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:28.992492 master-1 kubenswrapper[4771]: I1011 10:55:28.980176 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-8vzsw"] Oct 11 10:55:29.140068 master-1 kubenswrapper[4771]: I1011 10:55:29.139757 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-catalog-content\") pod \"redhat-operators-8vzsw\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:29.140396 master-1 kubenswrapper[4771]: I1011 10:55:29.140095 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sl2p\" (UniqueName: \"kubernetes.io/projected/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-kube-api-access-8sl2p\") pod \"redhat-operators-8vzsw\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:29.140396 master-1 kubenswrapper[4771]: I1011 10:55:29.140178 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-utilities\") pod \"redhat-operators-8vzsw\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:29.244402 master-1 kubenswrapper[4771]: I1011 10:55:29.242671 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-catalog-content\") pod \"redhat-operators-8vzsw\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:29.244402 master-1 kubenswrapper[4771]: I1011 10:55:29.242738 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl2p\" (UniqueName: \"kubernetes.io/projected/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-kube-api-access-8sl2p\") pod \"redhat-operators-8vzsw\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:29.244402 master-1 kubenswrapper[4771]: I1011 10:55:29.242792 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-utilities\") pod \"redhat-operators-8vzsw\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:29.244402 master-1 kubenswrapper[4771]: I1011 10:55:29.243313 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-utilities\") pod \"redhat-operators-8vzsw\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:29.244402 master-1 kubenswrapper[4771]: I1011 10:55:29.243608 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-catalog-content\") pod \"redhat-operators-8vzsw\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:29.294388 master-1 kubenswrapper[4771]: I1011 10:55:29.293630 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sl2p\" (UniqueName: \"kubernetes.io/projected/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-kube-api-access-8sl2p\") pod \"redhat-operators-8vzsw\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:29.298377 master-1 kubenswrapper[4771]: I1011 10:55:29.298074 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:31.462646 master-1 kubenswrapper[4771]: I1011 10:55:31.462595 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:55:31.550952 master-1 kubenswrapper[4771]: I1011 10:55:31.550878 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-595686b98f-blmgp"] Oct 11 10:55:31.551268 master-1 kubenswrapper[4771]: I1011 10:55:31.551180 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/dnsmasq-dns-595686b98f-blmgp" podUID="a50b2fec-a3b6-4245-9080-5987b411b581" containerName="dnsmasq-dns" containerID="cri-o://64768fb3aaa57fbf977b42bcf01d911517cd3d56cc20742d472651a90c1c3f06" gracePeriod=10 Oct 11 10:55:31.978958 master-1 kubenswrapper[4771]: I1011 10:55:31.978896 4771 generic.go:334] "Generic (PLEG): container finished" podID="a50b2fec-a3b6-4245-9080-5987b411b581" containerID="64768fb3aaa57fbf977b42bcf01d911517cd3d56cc20742d472651a90c1c3f06" exitCode=0 Oct 11 10:55:31.979465 master-1 kubenswrapper[4771]: I1011 10:55:31.978986 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-595686b98f-blmgp" event={"ID":"a50b2fec-a3b6-4245-9080-5987b411b581","Type":"ContainerDied","Data":"64768fb3aaa57fbf977b42bcf01d911517cd3d56cc20742d472651a90c1c3f06"} Oct 11 10:55:32.295450 master-1 kubenswrapper[4771]: I1011 10:55:32.295233 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/cinder-b5802-api-0" Oct 11 10:55:32.486745 master-1 kubenswrapper[4771]: I1011 10:55:32.486679 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:32.487342 master-1 kubenswrapper[4771]: I1011 10:55:32.486765 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:32.537275 master-1 kubenswrapper[4771]: I1011 10:55:32.537207 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:32.538419 master-1 kubenswrapper[4771]: I1011 10:55:32.538384 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:32.933087 master-1 kubenswrapper[4771]: I1011 10:55:32.932647 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/dnsmasq-dns-595686b98f-blmgp" podUID="a50b2fec-a3b6-4245-9080-5987b411b581" containerName="dnsmasq-dns" probeResult="failure" output="dial tcp 10.129.0.124:5353: connect: connection refused" Oct 11 10:55:32.991118 master-1 kubenswrapper[4771]: I1011 10:55:32.990933 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:32.991118 master-1 kubenswrapper[4771]: I1011 10:55:32.991024 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:34.781391 master-1 kubenswrapper[4771]: I1011 10:55:34.781232 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:34.931685 master-1 kubenswrapper[4771]: I1011 10:55:34.931629 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:35.016712 master-1 kubenswrapper[4771]: I1011 10:55:35.016599 4771 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Oct 11 10:55:35.072657 master-1 kubenswrapper[4771]: I1011 10:55:35.072464 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:55:36.015439 master-1 kubenswrapper[4771]: I1011 10:55:36.013805 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:55:36.030793 master-1 kubenswrapper[4771]: I1011 10:55:36.030750 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-595686b98f-blmgp" Oct 11 10:55:36.030986 master-1 kubenswrapper[4771]: I1011 10:55:36.030838 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-595686b98f-blmgp" event={"ID":"a50b2fec-a3b6-4245-9080-5987b411b581","Type":"ContainerDied","Data":"df3edeb105ed637b9d7fa0933dc5cae9f70ee8feff1cdbfb3585b9bc6889a72c"} Oct 11 10:55:36.030986 master-1 kubenswrapper[4771]: I1011 10:55:36.030960 4771 scope.go:117] "RemoveContainer" containerID="64768fb3aaa57fbf977b42bcf01d911517cd3d56cc20742d472651a90c1c3f06" Oct 11 10:55:36.158475 master-1 kubenswrapper[4771]: I1011 10:55:36.158265 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-sb\") pod \"a50b2fec-a3b6-4245-9080-5987b411b581\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " Oct 11 10:55:36.158745 master-1 kubenswrapper[4771]: I1011 10:55:36.158534 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-swift-storage-0\") pod \"a50b2fec-a3b6-4245-9080-5987b411b581\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " Oct 11 10:55:36.158745 master-1 kubenswrapper[4771]: I1011 10:55:36.158683 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-svc\") pod \"a50b2fec-a3b6-4245-9080-5987b411b581\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " Oct 11 10:55:36.158745 master-1 kubenswrapper[4771]: I1011 10:55:36.158728 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-nb\") pod \"a50b2fec-a3b6-4245-9080-5987b411b581\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " Oct 11 10:55:36.158865 master-1 kubenswrapper[4771]: I1011 10:55:36.158776 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-config\") pod \"a50b2fec-a3b6-4245-9080-5987b411b581\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " Oct 11 10:55:36.158865 master-1 kubenswrapper[4771]: I1011 10:55:36.158830 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtcx9\" (UniqueName: \"kubernetes.io/projected/a50b2fec-a3b6-4245-9080-5987b411b581-kube-api-access-mtcx9\") pod \"a50b2fec-a3b6-4245-9080-5987b411b581\" (UID: \"a50b2fec-a3b6-4245-9080-5987b411b581\") " Oct 11 10:55:36.165135 master-1 kubenswrapper[4771]: I1011 10:55:36.165061 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50b2fec-a3b6-4245-9080-5987b411b581-kube-api-access-mtcx9" (OuterVolumeSpecName: "kube-api-access-mtcx9") pod "a50b2fec-a3b6-4245-9080-5987b411b581" (UID: "a50b2fec-a3b6-4245-9080-5987b411b581"). InnerVolumeSpecName "kube-api-access-mtcx9". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:36.217041 master-1 kubenswrapper[4771]: I1011 10:55:36.216897 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-swift-storage-0" (OuterVolumeSpecName: "dns-swift-storage-0") pod "a50b2fec-a3b6-4245-9080-5987b411b581" (UID: "a50b2fec-a3b6-4245-9080-5987b411b581"). InnerVolumeSpecName "dns-swift-storage-0". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:55:36.220856 master-1 kubenswrapper[4771]: I1011 10:55:36.220459 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-nb" (OuterVolumeSpecName: "ovsdbserver-nb") pod "a50b2fec-a3b6-4245-9080-5987b411b581" (UID: "a50b2fec-a3b6-4245-9080-5987b411b581"). InnerVolumeSpecName "ovsdbserver-nb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:55:36.224725 master-1 kubenswrapper[4771]: I1011 10:55:36.224612 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "a50b2fec-a3b6-4245-9080-5987b411b581" (UID: "a50b2fec-a3b6-4245-9080-5987b411b581"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:55:36.229991 master-1 kubenswrapper[4771]: I1011 10:55:36.229923 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-config" (OuterVolumeSpecName: "config") pod "a50b2fec-a3b6-4245-9080-5987b411b581" (UID: "a50b2fec-a3b6-4245-9080-5987b411b581"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:55:36.238989 master-1 kubenswrapper[4771]: I1011 10:55:36.238921 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-sb" (OuterVolumeSpecName: "ovsdbserver-sb") pod "a50b2fec-a3b6-4245-9080-5987b411b581" (UID: "a50b2fec-a3b6-4245-9080-5987b411b581"). InnerVolumeSpecName "ovsdbserver-sb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:55:36.262526 master-1 kubenswrapper[4771]: I1011 10:55:36.262453 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-sb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:36.262526 master-1 kubenswrapper[4771]: I1011 10:55:36.262510 4771 reconciler_common.go:293] "Volume detached for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-swift-storage-0\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:36.262526 master-1 kubenswrapper[4771]: I1011 10:55:36.262530 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:36.262526 master-1 kubenswrapper[4771]: I1011 10:55:36.262541 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-ovsdbserver-nb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:36.263012 master-1 kubenswrapper[4771]: I1011 10:55:36.262552 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a50b2fec-a3b6-4245-9080-5987b411b581-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:36.263012 master-1 kubenswrapper[4771]: I1011 10:55:36.262563 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-mtcx9\" (UniqueName: \"kubernetes.io/projected/a50b2fec-a3b6-4245-9080-5987b411b581-kube-api-access-mtcx9\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:36.340923 master-1 kubenswrapper[4771]: I1011 10:55:36.340867 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/heat-cfnapi-64fcdf7d54-8r455"] Oct 11 10:55:36.367981 master-1 kubenswrapper[4771]: W1011 10:55:36.367867 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dc94855_37f8_4fa8_a3e1_72808b37f966.slice/crio-7aec9cb2b97e7d8a103a161accbb44e46472e2bd0c84ba74baf43bde7dc083ff WatchSource:0}: Error finding container 7aec9cb2b97e7d8a103a161accbb44e46472e2bd0c84ba74baf43bde7dc083ff: Status 404 returned error can't find the container with id 7aec9cb2b97e7d8a103a161accbb44e46472e2bd0c84ba74baf43bde7dc083ff Oct 11 10:55:36.390452 master-1 kubenswrapper[4771]: I1011 10:55:36.390338 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-595686b98f-blmgp"] Oct 11 10:55:36.402471 master-1 kubenswrapper[4771]: I1011 10:55:36.402374 4771 scope.go:117] "RemoveContainer" containerID="314a76b2857d795a4f3ebe7e8b09e8abca5d105e5ba862e3833d60a9a90b7cc3" Oct 11 10:55:36.403085 master-1 kubenswrapper[4771]: I1011 10:55:36.402948 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-595686b98f-blmgp"] Oct 11 10:55:36.491034 master-1 kubenswrapper[4771]: I1011 10:55:36.490925 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a50b2fec-a3b6-4245-9080-5987b411b581" path="/var/lib/kubelet/pods/a50b2fec-a3b6-4245-9080-5987b411b581/volumes" Oct 11 10:55:36.824953 master-1 kubenswrapper[4771]: I1011 10:55:36.824886 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-8vzsw"] Oct 11 10:55:37.044189 master-1 kubenswrapper[4771]: I1011 10:55:37.043207 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-7cddc977f5-9ddgm" event={"ID":"879970ca-6312-4aec-b8f4-a8a41a0e3797","Type":"ContainerStarted","Data":"ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104"} Oct 11 10:55:37.047593 master-1 kubenswrapper[4771]: I1011 10:55:37.044944 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-8vzsw" event={"ID":"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44","Type":"ContainerStarted","Data":"e046736cf54a6f375a2d21055bc37323ff6d218a499c8b0059aa035f5e4d1a0c"} Oct 11 10:55:37.047593 master-1 kubenswrapper[4771]: I1011 10:55:37.045035 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-8vzsw" event={"ID":"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44","Type":"ContainerStarted","Data":"2e03e915f6f95ecc8f0f52052466e21bd1b0bb1a12eb203399bd0345ac65bccf"} Oct 11 10:55:37.047593 master-1 kubenswrapper[4771]: I1011 10:55:37.047126 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerStarted","Data":"10573443fa9f81c261e267c2d4f01ad7d7cf7482785a8f4f22c2ccd3fa1fc631"} Oct 11 10:55:37.050486 master-1 kubenswrapper[4771]: I1011 10:55:37.049978 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" event={"ID":"c6af8eba-f8bf-47f6-8313-7a902aeb170f","Type":"ContainerStarted","Data":"18f55509d99d6df6e062c9f98f3f97b0989b5f829acb0a772e9a836bc344b833"} Oct 11 10:55:37.050486 master-1 kubenswrapper[4771]: I1011 10:55:37.050200 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:37.051892 master-1 kubenswrapper[4771]: I1011 10:55:37.051865 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" event={"ID":"1fe7833d-9251-4545-ba68-f58c146188f1","Type":"ContainerStarted","Data":"3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d"} Oct 11 10:55:37.052015 master-1 kubenswrapper[4771]: I1011 10:55:37.051989 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" podUID="1fe7833d-9251-4545-ba68-f58c146188f1" containerName="heat-cfnapi" containerID="cri-o://3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d" gracePeriod=60 Oct 11 10:55:37.052288 master-1 kubenswrapper[4771]: I1011 10:55:37.052269 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:37.054945 master-1 kubenswrapper[4771]: I1011 10:55:37.054916 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-cfnapi-64fcdf7d54-8r455" event={"ID":"2dc94855-37f8-4fa8-a3e1-72808b37f966","Type":"ContainerStarted","Data":"3d839e5a2690f36d8e0304ba13a497d4c3537f02e2e7261be9b1a7abfcf45c44"} Oct 11 10:55:37.054945 master-1 kubenswrapper[4771]: I1011 10:55:37.054946 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-cfnapi-64fcdf7d54-8r455" event={"ID":"2dc94855-37f8-4fa8-a3e1-72808b37f966","Type":"ContainerStarted","Data":"7aec9cb2b97e7d8a103a161accbb44e46472e2bd0c84ba74baf43bde7dc083ff"} Oct 11 10:55:37.055589 master-1 kubenswrapper[4771]: I1011 10:55:37.055551 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:37.057154 master-1 kubenswrapper[4771]: I1011 10:55:37.057110 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-api-d647f9c47-x7xc2" event={"ID":"e0657ee5-2e60-4a96-905e-814f46a72970","Type":"ContainerStarted","Data":"a5ba999f8e1551f739b0074644873dac43d11fc22bc8e7bb8107aced2b4ca581"} Oct 11 10:55:37.057496 master-1 kubenswrapper[4771]: I1011 10:55:37.057474 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:37.090349 master-1 kubenswrapper[4771]: I1011 10:55:37.090251 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" podStartSLOduration=11.098397867 podStartE2EDuration="23.090223872s" podCreationTimestamp="2025-10-11 10:55:14 +0000 UTC" firstStartedPulling="2025-10-11 10:55:23.923328249 +0000 UTC m=+1755.897554700" lastFinishedPulling="2025-10-11 10:55:35.915154264 +0000 UTC m=+1767.889380705" observedRunningTime="2025-10-11 10:55:37.09012523 +0000 UTC m=+1769.064351691" watchObservedRunningTime="2025-10-11 10:55:37.090223872 +0000 UTC m=+1769.064450313" Oct 11 10:55:37.120533 master-1 kubenswrapper[4771]: I1011 10:55:37.120433 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" podStartSLOduration=8.200960996 podStartE2EDuration="21.120406253s" podCreationTimestamp="2025-10-11 10:55:16 +0000 UTC" firstStartedPulling="2025-10-11 10:55:23.508151501 +0000 UTC m=+1755.482377942" lastFinishedPulling="2025-10-11 10:55:36.427596758 +0000 UTC m=+1768.401823199" observedRunningTime="2025-10-11 10:55:37.112823812 +0000 UTC m=+1769.087050243" watchObservedRunningTime="2025-10-11 10:55:37.120406253 +0000 UTC m=+1769.094632694" Oct 11 10:55:37.153108 master-1 kubenswrapper[4771]: I1011 10:55:37.153009 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/heat-api-d647f9c47-x7xc2" podStartSLOduration=2.433012732 podStartE2EDuration="12.152975062s" podCreationTimestamp="2025-10-11 10:55:25 +0000 UTC" firstStartedPulling="2025-10-11 10:55:26.733145152 +0000 UTC m=+1758.707371593" lastFinishedPulling="2025-10-11 10:55:36.453107472 +0000 UTC m=+1768.427333923" observedRunningTime="2025-10-11 10:55:37.146910906 +0000 UTC m=+1769.121137357" watchObservedRunningTime="2025-10-11 10:55:37.152975062 +0000 UTC m=+1769.127201503" Oct 11 10:55:37.179023 master-1 kubenswrapper[4771]: I1011 10:55:37.178936 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/heat-cfnapi-64fcdf7d54-8r455" podStartSLOduration=10.178915539 podStartE2EDuration="10.178915539s" podCreationTimestamp="2025-10-11 10:55:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:55:37.176364765 +0000 UTC m=+1769.150591216" watchObservedRunningTime="2025-10-11 10:55:37.178915539 +0000 UTC m=+1769.153141980" Oct 11 10:55:38.068402 master-1 kubenswrapper[4771]: I1011 10:55:38.068206 4771 generic.go:334] "Generic (PLEG): container finished" podID="e0657ee5-2e60-4a96-905e-814f46a72970" containerID="a5ba999f8e1551f739b0074644873dac43d11fc22bc8e7bb8107aced2b4ca581" exitCode=1 Oct 11 10:55:38.068402 master-1 kubenswrapper[4771]: I1011 10:55:38.068274 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-api-d647f9c47-x7xc2" event={"ID":"e0657ee5-2e60-4a96-905e-814f46a72970","Type":"ContainerDied","Data":"a5ba999f8e1551f739b0074644873dac43d11fc22bc8e7bb8107aced2b4ca581"} Oct 11 10:55:38.069459 master-1 kubenswrapper[4771]: I1011 10:55:38.069286 4771 scope.go:117] "RemoveContainer" containerID="a5ba999f8e1551f739b0074644873dac43d11fc22bc8e7bb8107aced2b4ca581" Oct 11 10:55:38.072735 master-1 kubenswrapper[4771]: I1011 10:55:38.072629 4771 generic.go:334] "Generic (PLEG): container finished" podID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerID="ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104" exitCode=0 Oct 11 10:55:38.072957 master-1 kubenswrapper[4771]: I1011 10:55:38.072732 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-7cddc977f5-9ddgm" event={"ID":"879970ca-6312-4aec-b8f4-a8a41a0e3797","Type":"ContainerDied","Data":"ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104"} Oct 11 10:55:38.076961 master-1 kubenswrapper[4771]: I1011 10:55:38.076903 4771 generic.go:334] "Generic (PLEG): container finished" podID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerID="e046736cf54a6f375a2d21055bc37323ff6d218a499c8b0059aa035f5e4d1a0c" exitCode=0 Oct 11 10:55:38.077055 master-1 kubenswrapper[4771]: I1011 10:55:38.077013 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-8vzsw" event={"ID":"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44","Type":"ContainerDied","Data":"e046736cf54a6f375a2d21055bc37323ff6d218a499c8b0059aa035f5e4d1a0c"} Oct 11 10:55:38.917865 master-1 kubenswrapper[4771]: I1011 10:55:38.917229 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/heat-api-d647f9c47-x7xc2"] Oct 11 10:55:39.091128 master-1 kubenswrapper[4771]: I1011 10:55:39.090884 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-8vzsw" event={"ID":"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44","Type":"ContainerStarted","Data":"f4caf5c874767c7fd75c2e84ff37e1b5c988f50fb776ae2062994f2e951ecc23"} Oct 11 10:55:39.095895 master-1 kubenswrapper[4771]: I1011 10:55:39.095548 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerStarted","Data":"7f100b006260b4ff812a662ca4646172d63077e3423c5b53974c9a4fc93bb108"} Oct 11 10:55:39.096093 master-1 kubenswrapper[4771]: I1011 10:55:39.095970 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 10:55:39.096093 master-1 kubenswrapper[4771]: I1011 10:55:39.096020 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="proxy-httpd" containerID="cri-o://7f100b006260b4ff812a662ca4646172d63077e3423c5b53974c9a4fc93bb108" gracePeriod=30 Oct 11 10:55:39.096434 master-1 kubenswrapper[4771]: I1011 10:55:39.095967 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="ceilometer-notification-agent" containerID="cri-o://1d0d93b3fc6393dcdc851e8c3921d7c5d5a44cf9e99d331f9e66f61b3c48f59d" gracePeriod=30 Oct 11 10:55:39.096434 master-1 kubenswrapper[4771]: I1011 10:55:39.095979 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="sg-core" containerID="cri-o://10573443fa9f81c261e267c2d4f01ad7d7cf7482785a8f4f22c2ccd3fa1fc631" gracePeriod=30 Oct 11 10:55:39.096434 master-1 kubenswrapper[4771]: I1011 10:55:39.095947 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="ceilometer-central-agent" containerID="cri-o://33e1159e64df7103066e5f7850051b2adc3d09e823478d0dc1137ddef2aee326" gracePeriod=30 Oct 11 10:55:39.100007 master-1 kubenswrapper[4771]: I1011 10:55:39.098777 4771 generic.go:334] "Generic (PLEG): container finished" podID="e0657ee5-2e60-4a96-905e-814f46a72970" containerID="359e13273e98466e823fa5c4d2aba3d9afd810ad691524f34525677325371beb" exitCode=1 Oct 11 10:55:39.100007 master-1 kubenswrapper[4771]: I1011 10:55:39.098853 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-api-d647f9c47-x7xc2" event={"ID":"e0657ee5-2e60-4a96-905e-814f46a72970","Type":"ContainerDied","Data":"359e13273e98466e823fa5c4d2aba3d9afd810ad691524f34525677325371beb"} Oct 11 10:55:39.100007 master-1 kubenswrapper[4771]: I1011 10:55:39.098919 4771 scope.go:117] "RemoveContainer" containerID="a5ba999f8e1551f739b0074644873dac43d11fc22bc8e7bb8107aced2b4ca581" Oct 11 10:55:39.102479 master-1 kubenswrapper[4771]: I1011 10:55:39.102416 4771 scope.go:117] "RemoveContainer" containerID="359e13273e98466e823fa5c4d2aba3d9afd810ad691524f34525677325371beb" Oct 11 10:55:39.103968 master-1 kubenswrapper[4771]: I1011 10:55:39.103881 4771 generic.go:334] "Generic (PLEG): container finished" podID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerID="47d878804e10bb973f86c4fb2c2dafde704a472be7b960d95d71befafe9306e4" exitCode=1 Oct 11 10:55:39.103968 master-1 kubenswrapper[4771]: I1011 10:55:39.103929 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-7cddc977f5-9ddgm" event={"ID":"879970ca-6312-4aec-b8f4-a8a41a0e3797","Type":"ContainerDied","Data":"47d878804e10bb973f86c4fb2c2dafde704a472be7b960d95d71befafe9306e4"} Oct 11 10:55:39.103968 master-1 kubenswrapper[4771]: I1011 10:55:39.103952 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-7cddc977f5-9ddgm" event={"ID":"879970ca-6312-4aec-b8f4-a8a41a0e3797","Type":"ContainerStarted","Data":"a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e"} Oct 11 10:55:39.105137 master-1 kubenswrapper[4771]: E1011 10:55:39.105067 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"heat-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=heat-api pod=heat-api-d647f9c47-x7xc2_openstack(e0657ee5-2e60-4a96-905e-814f46a72970)\"" pod="openstack/heat-api-d647f9c47-x7xc2" podUID="e0657ee5-2e60-4a96-905e-814f46a72970" Oct 11 10:55:39.105208 master-1 kubenswrapper[4771]: I1011 10:55:39.105128 4771 scope.go:117] "RemoveContainer" containerID="47d878804e10bb973f86c4fb2c2dafde704a472be7b960d95d71befafe9306e4" Oct 11 10:55:39.187341 master-1 kubenswrapper[4771]: I1011 10:55:39.186918 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=2.900443888 podStartE2EDuration="16.186896627s" podCreationTimestamp="2025-10-11 10:55:23 +0000 UTC" firstStartedPulling="2025-10-11 10:55:24.729574292 +0000 UTC m=+1756.703800733" lastFinishedPulling="2025-10-11 10:55:38.016027001 +0000 UTC m=+1769.990253472" observedRunningTime="2025-10-11 10:55:39.183158868 +0000 UTC m=+1771.157385319" watchObservedRunningTime="2025-10-11 10:55:39.186896627 +0000 UTC m=+1771.161123078" Oct 11 10:55:40.117039 master-1 kubenswrapper[4771]: I1011 10:55:40.116984 4771 generic.go:334] "Generic (PLEG): container finished" podID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerID="f4caf5c874767c7fd75c2e84ff37e1b5c988f50fb776ae2062994f2e951ecc23" exitCode=0 Oct 11 10:55:40.117930 master-1 kubenswrapper[4771]: I1011 10:55:40.117060 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-8vzsw" event={"ID":"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44","Type":"ContainerDied","Data":"f4caf5c874767c7fd75c2e84ff37e1b5c988f50fb776ae2062994f2e951ecc23"} Oct 11 10:55:40.121170 master-1 kubenswrapper[4771]: I1011 10:55:40.121113 4771 generic.go:334] "Generic (PLEG): container finished" podID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerID="7f100b006260b4ff812a662ca4646172d63077e3423c5b53974c9a4fc93bb108" exitCode=0 Oct 11 10:55:40.121170 master-1 kubenswrapper[4771]: I1011 10:55:40.121158 4771 generic.go:334] "Generic (PLEG): container finished" podID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerID="10573443fa9f81c261e267c2d4f01ad7d7cf7482785a8f4f22c2ccd3fa1fc631" exitCode=2 Oct 11 10:55:40.121170 master-1 kubenswrapper[4771]: I1011 10:55:40.121168 4771 generic.go:334] "Generic (PLEG): container finished" podID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerID="33e1159e64df7103066e5f7850051b2adc3d09e823478d0dc1137ddef2aee326" exitCode=0 Oct 11 10:55:40.121428 master-1 kubenswrapper[4771]: I1011 10:55:40.121216 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerDied","Data":"7f100b006260b4ff812a662ca4646172d63077e3423c5b53974c9a4fc93bb108"} Oct 11 10:55:40.121428 master-1 kubenswrapper[4771]: I1011 10:55:40.121309 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerDied","Data":"10573443fa9f81c261e267c2d4f01ad7d7cf7482785a8f4f22c2ccd3fa1fc631"} Oct 11 10:55:40.121428 master-1 kubenswrapper[4771]: I1011 10:55:40.121334 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerDied","Data":"33e1159e64df7103066e5f7850051b2adc3d09e823478d0dc1137ddef2aee326"} Oct 11 10:55:40.123436 master-1 kubenswrapper[4771]: I1011 10:55:40.123376 4771 generic.go:334] "Generic (PLEG): container finished" podID="c6af8eba-f8bf-47f6-8313-7a902aeb170f" containerID="18f55509d99d6df6e062c9f98f3f97b0989b5f829acb0a772e9a836bc344b833" exitCode=1 Oct 11 10:55:40.123579 master-1 kubenswrapper[4771]: I1011 10:55:40.123474 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" event={"ID":"c6af8eba-f8bf-47f6-8313-7a902aeb170f","Type":"ContainerDied","Data":"18f55509d99d6df6e062c9f98f3f97b0989b5f829acb0a772e9a836bc344b833"} Oct 11 10:55:40.123899 master-1 kubenswrapper[4771]: I1011 10:55:40.123856 4771 scope.go:117] "RemoveContainer" containerID="18f55509d99d6df6e062c9f98f3f97b0989b5f829acb0a772e9a836bc344b833" Oct 11 10:55:40.137902 master-1 kubenswrapper[4771]: I1011 10:55:40.137836 4771 generic.go:334] "Generic (PLEG): container finished" podID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerID="d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb" exitCode=1 Oct 11 10:55:40.137902 master-1 kubenswrapper[4771]: I1011 10:55:40.137890 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-7cddc977f5-9ddgm" event={"ID":"879970ca-6312-4aec-b8f4-a8a41a0e3797","Type":"ContainerDied","Data":"d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb"} Oct 11 10:55:40.138171 master-1 kubenswrapper[4771]: I1011 10:55:40.137933 4771 scope.go:117] "RemoveContainer" containerID="47d878804e10bb973f86c4fb2c2dafde704a472be7b960d95d71befafe9306e4" Oct 11 10:55:40.138947 master-1 kubenswrapper[4771]: I1011 10:55:40.138919 4771 scope.go:117] "RemoveContainer" containerID="d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb" Oct 11 10:55:40.139229 master-1 kubenswrapper[4771]: E1011 10:55:40.139201 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"ironic-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=ironic-api pod=ironic-7cddc977f5-9ddgm_openstack(879970ca-6312-4aec-b8f4-a8a41a0e3797)\"" pod="openstack/ironic-7cddc977f5-9ddgm" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" Oct 11 10:55:40.277338 master-1 kubenswrapper[4771]: I1011 10:55:40.277294 4771 kubelet.go:2542] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:40.539122 master-1 kubenswrapper[4771]: I1011 10:55:40.539053 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:40.585984 master-1 kubenswrapper[4771]: I1011 10:55:40.585905 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:40.586402 master-1 kubenswrapper[4771]: I1011 10:55:40.586019 4771 kubelet.go:2542] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:40.595078 master-1 kubenswrapper[4771]: I1011 10:55:40.595020 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-combined-ca-bundle\") pod \"e0657ee5-2e60-4a96-905e-814f46a72970\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " Oct 11 10:55:40.595266 master-1 kubenswrapper[4771]: I1011 10:55:40.595167 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data-custom\") pod \"e0657ee5-2e60-4a96-905e-814f46a72970\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " Oct 11 10:55:40.595266 master-1 kubenswrapper[4771]: I1011 10:55:40.595225 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data\") pod \"e0657ee5-2e60-4a96-905e-814f46a72970\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " Oct 11 10:55:40.595437 master-1 kubenswrapper[4771]: I1011 10:55:40.595384 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z885b\" (UniqueName: \"kubernetes.io/projected/e0657ee5-2e60-4a96-905e-814f46a72970-kube-api-access-z885b\") pod \"e0657ee5-2e60-4a96-905e-814f46a72970\" (UID: \"e0657ee5-2e60-4a96-905e-814f46a72970\") " Oct 11 10:55:40.611252 master-1 kubenswrapper[4771]: I1011 10:55:40.603574 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data-custom" (OuterVolumeSpecName: "config-data-custom") pod "e0657ee5-2e60-4a96-905e-814f46a72970" (UID: "e0657ee5-2e60-4a96-905e-814f46a72970"). InnerVolumeSpecName "config-data-custom". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:40.611252 master-1 kubenswrapper[4771]: I1011 10:55:40.603574 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0657ee5-2e60-4a96-905e-814f46a72970-kube-api-access-z885b" (OuterVolumeSpecName: "kube-api-access-z885b") pod "e0657ee5-2e60-4a96-905e-814f46a72970" (UID: "e0657ee5-2e60-4a96-905e-814f46a72970"). InnerVolumeSpecName "kube-api-access-z885b". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:40.620132 master-1 kubenswrapper[4771]: I1011 10:55:40.620075 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "e0657ee5-2e60-4a96-905e-814f46a72970" (UID: "e0657ee5-2e60-4a96-905e-814f46a72970"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:40.649478 master-1 kubenswrapper[4771]: I1011 10:55:40.648699 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data" (OuterVolumeSpecName: "config-data") pod "e0657ee5-2e60-4a96-905e-814f46a72970" (UID: "e0657ee5-2e60-4a96-905e-814f46a72970"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:40.698071 master-1 kubenswrapper[4771]: I1011 10:55:40.698006 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-z885b\" (UniqueName: \"kubernetes.io/projected/e0657ee5-2e60-4a96-905e-814f46a72970-kube-api-access-z885b\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:40.698071 master-1 kubenswrapper[4771]: I1011 10:55:40.698064 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:40.698262 master-1 kubenswrapper[4771]: I1011 10:55:40.698082 4771 reconciler_common.go:293] "Volume detached for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data-custom\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:40.698262 master-1 kubenswrapper[4771]: I1011 10:55:40.698104 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e0657ee5-2e60-4a96-905e-814f46a72970-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:41.152248 master-1 kubenswrapper[4771]: I1011 10:55:41.152043 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" event={"ID":"c6af8eba-f8bf-47f6-8313-7a902aeb170f","Type":"ContainerStarted","Data":"d389be566649d008a3e09b2815ff87839b176862458587a6a08bb0703f09d204"} Oct 11 10:55:41.155345 master-1 kubenswrapper[4771]: I1011 10:55:41.155270 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-api-d647f9c47-x7xc2" event={"ID":"e0657ee5-2e60-4a96-905e-814f46a72970","Type":"ContainerDied","Data":"4d5653f9de27a6b9172b5e020cb9d04e796130d0328e34ab12c5dd9a66c1452e"} Oct 11 10:55:41.155528 master-1 kubenswrapper[4771]: I1011 10:55:41.155375 4771 scope.go:117] "RemoveContainer" containerID="359e13273e98466e823fa5c4d2aba3d9afd810ad691524f34525677325371beb" Oct 11 10:55:41.155608 master-1 kubenswrapper[4771]: I1011 10:55:41.155529 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/heat-api-d647f9c47-x7xc2" Oct 11 10:55:41.163397 master-1 kubenswrapper[4771]: I1011 10:55:41.163320 4771 scope.go:117] "RemoveContainer" containerID="d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb" Oct 11 10:55:41.163750 master-1 kubenswrapper[4771]: E1011 10:55:41.163707 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"ironic-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=ironic-api pod=ironic-7cddc977f5-9ddgm_openstack(879970ca-6312-4aec-b8f4-a8a41a0e3797)\"" pod="openstack/ironic-7cddc977f5-9ddgm" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" Oct 11 10:55:41.168366 master-1 kubenswrapper[4771]: I1011 10:55:41.168316 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-8vzsw" event={"ID":"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44","Type":"ContainerStarted","Data":"00b0738c103a1f8936204d6f12df3d6fd67321868af021b54204d55c141f77ca"} Oct 11 10:55:41.216412 master-1 kubenswrapper[4771]: I1011 10:55:41.216130 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-operators-8vzsw" podStartSLOduration=10.745220976 podStartE2EDuration="13.216107324s" podCreationTimestamp="2025-10-11 10:55:28 +0000 UTC" firstStartedPulling="2025-10-11 10:55:38.079524743 +0000 UTC m=+1770.053751224" lastFinishedPulling="2025-10-11 10:55:40.550411131 +0000 UTC m=+1772.524637572" observedRunningTime="2025-10-11 10:55:41.214205318 +0000 UTC m=+1773.188431769" watchObservedRunningTime="2025-10-11 10:55:41.216107324 +0000 UTC m=+1773.190333765" Oct 11 10:55:41.229194 master-1 kubenswrapper[4771]: I1011 10:55:41.229027 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/neutron-748bbfcf89-vpkvr" Oct 11 10:55:41.311379 master-1 kubenswrapper[4771]: I1011 10:55:41.311281 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/heat-api-d647f9c47-x7xc2"] Oct 11 10:55:41.326710 master-1 kubenswrapper[4771]: I1011 10:55:41.326257 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/heat-api-d647f9c47-x7xc2"] Oct 11 10:55:41.350550 master-1 kubenswrapper[4771]: I1011 10:55:41.350492 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/neutron-7887b79bcd-stzg5"] Oct 11 10:55:41.350876 master-1 kubenswrapper[4771]: I1011 10:55:41.350837 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/neutron-7887b79bcd-stzg5" podUID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerName="neutron-api" containerID="cri-o://b94dfe1997cbb3d378d19012a9b6401bc1cef35489c7ea7be575908bfe56b3a0" gracePeriod=30 Oct 11 10:55:41.351094 master-1 kubenswrapper[4771]: I1011 10:55:41.351043 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/neutron-7887b79bcd-stzg5" podUID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerName="neutron-httpd" containerID="cri-o://99502f3eb6699cc67bcf11374ee8446bc01a1a157ce8024301c91ebed596f3f2" gracePeriod=30 Oct 11 10:55:42.190122 master-1 kubenswrapper[4771]: I1011 10:55:42.190045 4771 generic.go:334] "Generic (PLEG): container finished" podID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerID="99502f3eb6699cc67bcf11374ee8446bc01a1a157ce8024301c91ebed596f3f2" exitCode=0 Oct 11 10:55:42.190693 master-1 kubenswrapper[4771]: I1011 10:55:42.190165 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-7887b79bcd-stzg5" event={"ID":"362d815c-c6ec-48b0-9891-85d06ad00aed","Type":"ContainerDied","Data":"99502f3eb6699cc67bcf11374ee8446bc01a1a157ce8024301c91ebed596f3f2"} Oct 11 10:55:42.193417 master-1 kubenswrapper[4771]: I1011 10:55:42.193342 4771 scope.go:117] "RemoveContainer" containerID="d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb" Oct 11 10:55:42.193820 master-1 kubenswrapper[4771]: E1011 10:55:42.193773 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"ironic-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=ironic-api pod=ironic-7cddc977f5-9ddgm_openstack(879970ca-6312-4aec-b8f4-a8a41a0e3797)\"" pod="openstack/ironic-7cddc977f5-9ddgm" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" Oct 11 10:55:42.195665 master-1 kubenswrapper[4771]: I1011 10:55:42.195607 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:42.452533 master-1 kubenswrapper[4771]: I1011 10:55:42.452283 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0657ee5-2e60-4a96-905e-814f46a72970" path="/var/lib/kubelet/pods/e0657ee5-2e60-4a96-905e-814f46a72970/volumes" Oct 11 10:55:42.755214 master-1 kubenswrapper[4771]: I1011 10:55:42.755111 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:55:43.217376 master-1 kubenswrapper[4771]: I1011 10:55:43.217299 4771 generic.go:334] "Generic (PLEG): container finished" podID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerID="1d0d93b3fc6393dcdc851e8c3921d7c5d5a44cf9e99d331f9e66f61b3c48f59d" exitCode=0 Oct 11 10:55:43.218129 master-1 kubenswrapper[4771]: I1011 10:55:43.217493 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerDied","Data":"1d0d93b3fc6393dcdc851e8c3921d7c5d5a44cf9e99d331f9e66f61b3c48f59d"} Oct 11 10:55:43.218129 master-1 kubenswrapper[4771]: I1011 10:55:43.217558 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"736a75e7-8c74-4862-9ac6-3b4c2d0d721d","Type":"ContainerDied","Data":"5c043b242e5c8ff65d3ea42d92bba671ec7f6446a265531a8e4be33feddbe4fa"} Oct 11 10:55:43.218129 master-1 kubenswrapper[4771]: I1011 10:55:43.217571 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c043b242e5c8ff65d3ea42d92bba671ec7f6446a265531a8e4be33feddbe4fa" Oct 11 10:55:43.256400 master-1 kubenswrapper[4771]: I1011 10:55:43.256227 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:43.299094 master-1 kubenswrapper[4771]: I1011 10:55:43.299046 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:43.492826 master-1 kubenswrapper[4771]: I1011 10:55:43.492650 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-run-httpd\") pod \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " Oct 11 10:55:43.492826 master-1 kubenswrapper[4771]: I1011 10:55:43.492763 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-combined-ca-bundle\") pod \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " Oct 11 10:55:43.492826 master-1 kubenswrapper[4771]: I1011 10:55:43.492793 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-sg-core-conf-yaml\") pod \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " Oct 11 10:55:43.492826 master-1 kubenswrapper[4771]: I1011 10:55:43.492815 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-scripts\") pod \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " Oct 11 10:55:43.493471 master-1 kubenswrapper[4771]: I1011 10:55:43.492938 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-log-httpd\") pod \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " Oct 11 10:55:43.493471 master-1 kubenswrapper[4771]: I1011 10:55:43.493084 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-config-data\") pod \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " Oct 11 10:55:43.493743 master-1 kubenswrapper[4771]: I1011 10:55:43.493708 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7f7t\" (UniqueName: \"kubernetes.io/projected/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-kube-api-access-v7f7t\") pod \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\" (UID: \"736a75e7-8c74-4862-9ac6-3b4c2d0d721d\") " Oct 11 10:55:43.493809 master-1 kubenswrapper[4771]: I1011 10:55:43.493094 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "736a75e7-8c74-4862-9ac6-3b4c2d0d721d" (UID: "736a75e7-8c74-4862-9ac6-3b4c2d0d721d"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:43.493893 master-1 kubenswrapper[4771]: I1011 10:55:43.493693 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "736a75e7-8c74-4862-9ac6-3b4c2d0d721d" (UID: "736a75e7-8c74-4862-9ac6-3b4c2d0d721d"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:43.494139 master-1 kubenswrapper[4771]: I1011 10:55:43.494109 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:43.494139 master-1 kubenswrapper[4771]: I1011 10:55:43.494136 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:43.512896 master-1 kubenswrapper[4771]: I1011 10:55:43.512684 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-kube-api-access-v7f7t" (OuterVolumeSpecName: "kube-api-access-v7f7t") pod "736a75e7-8c74-4862-9ac6-3b4c2d0d721d" (UID: "736a75e7-8c74-4862-9ac6-3b4c2d0d721d"). InnerVolumeSpecName "kube-api-access-v7f7t". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:43.513399 master-1 kubenswrapper[4771]: I1011 10:55:43.512953 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-scripts" (OuterVolumeSpecName: "scripts") pod "736a75e7-8c74-4862-9ac6-3b4c2d0d721d" (UID: "736a75e7-8c74-4862-9ac6-3b4c2d0d721d"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:43.529183 master-1 kubenswrapper[4771]: I1011 10:55:43.528714 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "736a75e7-8c74-4862-9ac6-3b4c2d0d721d" (UID: "736a75e7-8c74-4862-9ac6-3b4c2d0d721d"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:43.575018 master-1 kubenswrapper[4771]: I1011 10:55:43.574920 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "736a75e7-8c74-4862-9ac6-3b4c2d0d721d" (UID: "736a75e7-8c74-4862-9ac6-3b4c2d0d721d"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:43.591403 master-1 kubenswrapper[4771]: I1011 10:55:43.591311 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-config-data" (OuterVolumeSpecName: "config-data") pod "736a75e7-8c74-4862-9ac6-3b4c2d0d721d" (UID: "736a75e7-8c74-4862-9ac6-3b4c2d0d721d"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:43.596426 master-1 kubenswrapper[4771]: I1011 10:55:43.596373 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:43.596426 master-1 kubenswrapper[4771]: I1011 10:55:43.596411 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-v7f7t\" (UniqueName: \"kubernetes.io/projected/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-kube-api-access-v7f7t\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:43.596426 master-1 kubenswrapper[4771]: I1011 10:55:43.596425 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:43.596598 master-1 kubenswrapper[4771]: I1011 10:55:43.596435 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:43.596598 master-1 kubenswrapper[4771]: I1011 10:55:43.596446 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/736a75e7-8c74-4862-9ac6-3b4c2d0d721d-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:44.004800 master-1 kubenswrapper[4771]: I1011 10:55:44.004735 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/heat-cfnapi-64fcdf7d54-8r455" Oct 11 10:55:44.253029 master-1 kubenswrapper[4771]: I1011 10:55:44.252927 4771 generic.go:334] "Generic (PLEG): container finished" podID="c6af8eba-f8bf-47f6-8313-7a902aeb170f" containerID="d389be566649d008a3e09b2815ff87839b176862458587a6a08bb0703f09d204" exitCode=1 Oct 11 10:55:44.254146 master-1 kubenswrapper[4771]: I1011 10:55:44.253035 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" event={"ID":"c6af8eba-f8bf-47f6-8313-7a902aeb170f","Type":"ContainerDied","Data":"d389be566649d008a3e09b2815ff87839b176862458587a6a08bb0703f09d204"} Oct 11 10:55:44.254146 master-1 kubenswrapper[4771]: I1011 10:55:44.253129 4771 scope.go:117] "RemoveContainer" containerID="18f55509d99d6df6e062c9f98f3f97b0989b5f829acb0a772e9a836bc344b833" Oct 11 10:55:44.254146 master-1 kubenswrapper[4771]: I1011 10:55:44.253827 4771 scope.go:117] "RemoveContainer" containerID="d389be566649d008a3e09b2815ff87839b176862458587a6a08bb0703f09d204" Oct 11 10:55:44.254146 master-1 kubenswrapper[4771]: E1011 10:55:44.254106 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"ironic-neutron-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=ironic-neutron-agent pod=ironic-neutron-agent-656ddc8b67-kfkzr_openstack(c6af8eba-f8bf-47f6-8313-7a902aeb170f)\"" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" podUID="c6af8eba-f8bf-47f6-8313-7a902aeb170f" Oct 11 10:55:44.254146 master-1 kubenswrapper[4771]: I1011 10:55:44.254159 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:44.347001 master-1 kubenswrapper[4771]: I1011 10:55:44.346909 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:44.352830 master-1 kubenswrapper[4771]: I1011 10:55:44.352759 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:44.387200 master-1 kubenswrapper[4771]: I1011 10:55:44.387142 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:44.387562 master-1 kubenswrapper[4771]: E1011 10:55:44.387539 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e0657ee5-2e60-4a96-905e-814f46a72970" containerName="heat-api" Oct 11 10:55:44.387622 master-1 kubenswrapper[4771]: I1011 10:55:44.387564 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0657ee5-2e60-4a96-905e-814f46a72970" containerName="heat-api" Oct 11 10:55:44.387622 master-1 kubenswrapper[4771]: E1011 10:55:44.387585 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="ceilometer-central-agent" Oct 11 10:55:44.387622 master-1 kubenswrapper[4771]: I1011 10:55:44.387594 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="ceilometer-central-agent" Oct 11 10:55:44.387622 master-1 kubenswrapper[4771]: E1011 10:55:44.387616 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="proxy-httpd" Oct 11 10:55:44.387823 master-1 kubenswrapper[4771]: I1011 10:55:44.387625 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="proxy-httpd" Oct 11 10:55:44.387823 master-1 kubenswrapper[4771]: E1011 10:55:44.387656 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="sg-core" Oct 11 10:55:44.387823 master-1 kubenswrapper[4771]: I1011 10:55:44.387665 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="sg-core" Oct 11 10:55:44.387823 master-1 kubenswrapper[4771]: E1011 10:55:44.387684 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a50b2fec-a3b6-4245-9080-5987b411b581" containerName="init" Oct 11 10:55:44.387823 master-1 kubenswrapper[4771]: I1011 10:55:44.387692 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a50b2fec-a3b6-4245-9080-5987b411b581" containerName="init" Oct 11 10:55:44.387823 master-1 kubenswrapper[4771]: E1011 10:55:44.387712 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="ceilometer-notification-agent" Oct 11 10:55:44.387823 master-1 kubenswrapper[4771]: I1011 10:55:44.387721 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="ceilometer-notification-agent" Oct 11 10:55:44.387823 master-1 kubenswrapper[4771]: E1011 10:55:44.387740 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a50b2fec-a3b6-4245-9080-5987b411b581" containerName="dnsmasq-dns" Oct 11 10:55:44.387823 master-1 kubenswrapper[4771]: I1011 10:55:44.387748 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a50b2fec-a3b6-4245-9080-5987b411b581" containerName="dnsmasq-dns" Oct 11 10:55:44.388184 master-1 kubenswrapper[4771]: I1011 10:55:44.387934 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="sg-core" Oct 11 10:55:44.388184 master-1 kubenswrapper[4771]: I1011 10:55:44.387951 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e0657ee5-2e60-4a96-905e-814f46a72970" containerName="heat-api" Oct 11 10:55:44.388184 master-1 kubenswrapper[4771]: I1011 10:55:44.387968 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="proxy-httpd" Oct 11 10:55:44.388184 master-1 kubenswrapper[4771]: I1011 10:55:44.387982 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="ceilometer-central-agent" Oct 11 10:55:44.388184 master-1 kubenswrapper[4771]: I1011 10:55:44.388001 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" containerName="ceilometer-notification-agent" Oct 11 10:55:44.388184 master-1 kubenswrapper[4771]: I1011 10:55:44.388014 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a50b2fec-a3b6-4245-9080-5987b411b581" containerName="dnsmasq-dns" Oct 11 10:55:44.388545 master-1 kubenswrapper[4771]: E1011 10:55:44.388201 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e0657ee5-2e60-4a96-905e-814f46a72970" containerName="heat-api" Oct 11 10:55:44.388545 master-1 kubenswrapper[4771]: I1011 10:55:44.388213 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0657ee5-2e60-4a96-905e-814f46a72970" containerName="heat-api" Oct 11 10:55:44.388545 master-1 kubenswrapper[4771]: I1011 10:55:44.388530 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e0657ee5-2e60-4a96-905e-814f46a72970" containerName="heat-api" Oct 11 10:55:44.390791 master-1 kubenswrapper[4771]: I1011 10:55:44.390677 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:44.406143 master-1 kubenswrapper[4771]: I1011 10:55:44.406081 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 10:55:44.406655 master-1 kubenswrapper[4771]: I1011 10:55:44.406265 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 10:55:44.448059 master-1 kubenswrapper[4771]: I1011 10:55:44.447986 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="736a75e7-8c74-4862-9ac6-3b4c2d0d721d" path="/var/lib/kubelet/pods/736a75e7-8c74-4862-9ac6-3b4c2d0d721d/volumes" Oct 11 10:55:44.519379 master-1 kubenswrapper[4771]: I1011 10:55:44.519158 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl7xd\" (UniqueName: \"kubernetes.io/projected/0f6abca2-1aea-4da7-88aa-1d7651959165-kube-api-access-pl7xd\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.519379 master-1 kubenswrapper[4771]: I1011 10:55:44.519288 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-log-httpd\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.519379 master-1 kubenswrapper[4771]: I1011 10:55:44.519321 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-config-data\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.519736 master-1 kubenswrapper[4771]: I1011 10:55:44.519613 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.519736 master-1 kubenswrapper[4771]: I1011 10:55:44.519668 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-run-httpd\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.520157 master-1 kubenswrapper[4771]: I1011 10:55:44.520081 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.520217 master-1 kubenswrapper[4771]: I1011 10:55:44.520171 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-scripts\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.544578 master-1 kubenswrapper[4771]: I1011 10:55:44.544510 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:44.621433 master-1 kubenswrapper[4771]: I1011 10:55:44.621375 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.621723 master-1 kubenswrapper[4771]: I1011 10:55:44.621478 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-run-httpd\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.621723 master-1 kubenswrapper[4771]: I1011 10:55:44.621554 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.621723 master-1 kubenswrapper[4771]: I1011 10:55:44.621574 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-scripts\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.621723 master-1 kubenswrapper[4771]: I1011 10:55:44.621640 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-pl7xd\" (UniqueName: \"kubernetes.io/projected/0f6abca2-1aea-4da7-88aa-1d7651959165-kube-api-access-pl7xd\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.621723 master-1 kubenswrapper[4771]: I1011 10:55:44.621659 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-log-httpd\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.621723 master-1 kubenswrapper[4771]: I1011 10:55:44.621703 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-config-data\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.622872 master-1 kubenswrapper[4771]: I1011 10:55:44.622848 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-run-httpd\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.623005 master-1 kubenswrapper[4771]: I1011 10:55:44.622938 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-log-httpd\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.626235 master-1 kubenswrapper[4771]: I1011 10:55:44.626207 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-config-data\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.626322 master-1 kubenswrapper[4771]: I1011 10:55:44.626259 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.626913 master-1 kubenswrapper[4771]: I1011 10:55:44.626386 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.626913 master-1 kubenswrapper[4771]: I1011 10:55:44.626827 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-scripts\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.654421 master-1 kubenswrapper[4771]: I1011 10:55:44.654332 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl7xd\" (UniqueName: \"kubernetes.io/projected/0f6abca2-1aea-4da7-88aa-1d7651959165-kube-api-access-pl7xd\") pod \"ceilometer-0\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " pod="openstack/ceilometer-0" Oct 11 10:55:44.728934 master-1 kubenswrapper[4771]: I1011 10:55:44.728871 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:45.199074 master-1 kubenswrapper[4771]: I1011 10:55:45.198911 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:45.263825 master-1 kubenswrapper[4771]: I1011 10:55:45.263768 4771 scope.go:117] "RemoveContainer" containerID="d389be566649d008a3e09b2815ff87839b176862458587a6a08bb0703f09d204" Oct 11 10:55:45.264393 master-1 kubenswrapper[4771]: E1011 10:55:45.264076 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"ironic-neutron-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=ironic-neutron-agent pod=ironic-neutron-agent-656ddc8b67-kfkzr_openstack(c6af8eba-f8bf-47f6-8313-7a902aeb170f)\"" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" podUID="c6af8eba-f8bf-47f6-8313-7a902aeb170f" Oct 11 10:55:45.264447 master-1 kubenswrapper[4771]: I1011 10:55:45.264337 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerStarted","Data":"84ce7b87c3fe4abde506e09dd4965353c30550d9eae5707b3cd6ecad602405a9"} Oct 11 10:55:45.283558 master-1 kubenswrapper[4771]: I1011 10:55:45.283501 4771 kubelet.go:2542] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:45.283558 master-1 kubenswrapper[4771]: I1011 10:55:45.283539 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:55:46.275986 master-1 kubenswrapper[4771]: I1011 10:55:46.275828 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerStarted","Data":"927b471b1fa44ac0453124772c1c315e44a9bd395c3c5d07c01293a49d70a8ef"} Oct 11 10:55:46.277226 master-1 kubenswrapper[4771]: I1011 10:55:46.277172 4771 scope.go:117] "RemoveContainer" containerID="d389be566649d008a3e09b2815ff87839b176862458587a6a08bb0703f09d204" Oct 11 10:55:46.277605 master-1 kubenswrapper[4771]: E1011 10:55:46.277568 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"ironic-neutron-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=ironic-neutron-agent pod=ironic-neutron-agent-656ddc8b67-kfkzr_openstack(c6af8eba-f8bf-47f6-8313-7a902aeb170f)\"" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" podUID="c6af8eba-f8bf-47f6-8313-7a902aeb170f" Oct 11 10:55:46.530318 master-1 kubenswrapper[4771]: I1011 10:55:46.529635 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-api-a338-account-create-v4xzh"] Oct 11 10:55:46.531701 master-1 kubenswrapper[4771]: I1011 10:55:46.531668 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-a338-account-create-v4xzh" Oct 11 10:55:46.534823 master-1 kubenswrapper[4771]: I1011 10:55:46.534750 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-api-db-secret" Oct 11 10:55:46.563804 master-1 kubenswrapper[4771]: I1011 10:55:46.563665 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlsv\" (UniqueName: \"kubernetes.io/projected/7ac9af7f-afc6-4d4d-9923-db14ac820459-kube-api-access-8nlsv\") pod \"nova-api-a338-account-create-v4xzh\" (UID: \"7ac9af7f-afc6-4d4d-9923-db14ac820459\") " pod="openstack/nova-api-a338-account-create-v4xzh" Oct 11 10:55:46.665937 master-1 kubenswrapper[4771]: I1011 10:55:46.665841 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlsv\" (UniqueName: \"kubernetes.io/projected/7ac9af7f-afc6-4d4d-9923-db14ac820459-kube-api-access-8nlsv\") pod \"nova-api-a338-account-create-v4xzh\" (UID: \"7ac9af7f-afc6-4d4d-9923-db14ac820459\") " pod="openstack/nova-api-a338-account-create-v4xzh" Oct 11 10:55:46.694235 master-1 kubenswrapper[4771]: I1011 10:55:46.694164 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlsv\" (UniqueName: \"kubernetes.io/projected/7ac9af7f-afc6-4d4d-9923-db14ac820459-kube-api-access-8nlsv\") pod \"nova-api-a338-account-create-v4xzh\" (UID: \"7ac9af7f-afc6-4d4d-9923-db14ac820459\") " pod="openstack/nova-api-a338-account-create-v4xzh" Oct 11 10:55:46.864467 master-1 kubenswrapper[4771]: I1011 10:55:46.864399 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-api-a338-account-create-v4xzh"] Oct 11 10:55:46.907334 master-1 kubenswrapper[4771]: I1011 10:55:46.906521 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-a338-account-create-v4xzh" Oct 11 10:55:47.301389 master-1 kubenswrapper[4771]: I1011 10:55:47.301139 4771 generic.go:334] "Generic (PLEG): container finished" podID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerID="b94dfe1997cbb3d378d19012a9b6401bc1cef35489c7ea7be575908bfe56b3a0" exitCode=0 Oct 11 10:55:47.301389 master-1 kubenswrapper[4771]: I1011 10:55:47.301203 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-7887b79bcd-stzg5" event={"ID":"362d815c-c6ec-48b0-9891-85d06ad00aed","Type":"ContainerDied","Data":"b94dfe1997cbb3d378d19012a9b6401bc1cef35489c7ea7be575908bfe56b3a0"} Oct 11 10:55:47.305377 master-1 kubenswrapper[4771]: I1011 10:55:47.304109 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerStarted","Data":"e699798eb3d463b27b94ec4cd6a4bacb981768b5034628ba7f0e14e667c53445"} Oct 11 10:55:47.466387 master-1 kubenswrapper[4771]: I1011 10:55:47.465718 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-api-a338-account-create-v4xzh"] Oct 11 10:55:47.470610 master-1 kubenswrapper[4771]: W1011 10:55:47.468425 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ac9af7f_afc6_4d4d_9923_db14ac820459.slice/crio-9b12e8787780eb4c0e7ae7ec5652e12847e4d2cd1f0a158946584ba9e587475f WatchSource:0}: Error finding container 9b12e8787780eb4c0e7ae7ec5652e12847e4d2cd1f0a158946584ba9e587475f: Status 404 returned error can't find the container with id 9b12e8787780eb4c0e7ae7ec5652e12847e4d2cd1f0a158946584ba9e587475f Oct 11 10:55:47.699607 master-1 kubenswrapper[4771]: I1011 10:55:47.699507 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell0-conductor-db-sync-chcrd"] Oct 11 10:55:47.701333 master-1 kubenswrapper[4771]: I1011 10:55:47.701279 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:47.708149 master-1 kubenswrapper[4771]: I1011 10:55:47.708073 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell0-conductor-config-data" Oct 11 10:55:47.708616 master-1 kubenswrapper[4771]: I1011 10:55:47.708556 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell0-conductor-scripts" Oct 11 10:55:47.784062 master-1 kubenswrapper[4771]: I1011 10:55:47.783970 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell0-conductor-db-sync-chcrd"] Oct 11 10:55:47.907551 master-1 kubenswrapper[4771]: I1011 10:55:47.907487 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-config-data\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:47.907791 master-1 kubenswrapper[4771]: I1011 10:55:47.907585 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-combined-ca-bundle\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:47.907791 master-1 kubenswrapper[4771]: I1011 10:55:47.907636 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v6lx\" (UniqueName: \"kubernetes.io/projected/38267a66-0ebd-44ab-bc7f-cd5703503b74-kube-api-access-5v6lx\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:47.907791 master-1 kubenswrapper[4771]: I1011 10:55:47.907736 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-scripts\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:48.009181 master-1 kubenswrapper[4771]: I1011 10:55:48.009057 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-scripts\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:48.009181 master-1 kubenswrapper[4771]: I1011 10:55:48.009123 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-config-data\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:48.009181 master-1 kubenswrapper[4771]: I1011 10:55:48.009169 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-combined-ca-bundle\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:48.009556 master-1 kubenswrapper[4771]: I1011 10:55:48.009207 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5v6lx\" (UniqueName: \"kubernetes.io/projected/38267a66-0ebd-44ab-bc7f-cd5703503b74-kube-api-access-5v6lx\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:48.014310 master-1 kubenswrapper[4771]: I1011 10:55:48.014243 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-config-data\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:48.015291 master-1 kubenswrapper[4771]: I1011 10:55:48.015224 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-combined-ca-bundle\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:48.015817 master-1 kubenswrapper[4771]: I1011 10:55:48.015774 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-scripts\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:48.065851 master-1 kubenswrapper[4771]: I1011 10:55:48.065771 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v6lx\" (UniqueName: \"kubernetes.io/projected/38267a66-0ebd-44ab-bc7f-cd5703503b74-kube-api-access-5v6lx\") pod \"nova-cell0-conductor-db-sync-chcrd\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:48.323320 master-1 kubenswrapper[4771]: I1011 10:55:48.323150 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerStarted","Data":"dea9e948550286a77692eb6be22395f88e3b28cb639cfe7f80a137916ddb5ac1"} Oct 11 10:55:48.326075 master-1 kubenswrapper[4771]: I1011 10:55:48.325970 4771 generic.go:334] "Generic (PLEG): container finished" podID="7ac9af7f-afc6-4d4d-9923-db14ac820459" containerID="e1ee0992af169f3773493c300780fafe6521ac72bd4a220402d3338c4c92c6fb" exitCode=0 Oct 11 10:55:48.326075 master-1 kubenswrapper[4771]: I1011 10:55:48.326042 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-a338-account-create-v4xzh" event={"ID":"7ac9af7f-afc6-4d4d-9923-db14ac820459","Type":"ContainerDied","Data":"e1ee0992af169f3773493c300780fafe6521ac72bd4a220402d3338c4c92c6fb"} Oct 11 10:55:48.326335 master-1 kubenswrapper[4771]: I1011 10:55:48.326092 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-a338-account-create-v4xzh" event={"ID":"7ac9af7f-afc6-4d4d-9923-db14ac820459","Type":"ContainerStarted","Data":"9b12e8787780eb4c0e7ae7ec5652e12847e4d2cd1f0a158946584ba9e587475f"} Oct 11 10:55:48.327646 master-1 kubenswrapper[4771]: I1011 10:55:48.327553 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:55:49.155799 master-1 kubenswrapper[4771]: I1011 10:55:49.155719 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:49.261072 master-1 kubenswrapper[4771]: I1011 10:55:49.260867 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-config\") pod \"362d815c-c6ec-48b0-9891-85d06ad00aed\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " Oct 11 10:55:49.261072 master-1 kubenswrapper[4771]: I1011 10:55:49.261011 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-httpd-config\") pod \"362d815c-c6ec-48b0-9891-85d06ad00aed\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " Oct 11 10:55:49.261072 master-1 kubenswrapper[4771]: I1011 10:55:49.261064 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-ovndb-tls-certs\") pod \"362d815c-c6ec-48b0-9891-85d06ad00aed\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " Oct 11 10:55:49.261560 master-1 kubenswrapper[4771]: I1011 10:55:49.261198 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-combined-ca-bundle\") pod \"362d815c-c6ec-48b0-9891-85d06ad00aed\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " Oct 11 10:55:49.261560 master-1 kubenswrapper[4771]: I1011 10:55:49.261283 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4zt7\" (UniqueName: \"kubernetes.io/projected/362d815c-c6ec-48b0-9891-85d06ad00aed-kube-api-access-n4zt7\") pod \"362d815c-c6ec-48b0-9891-85d06ad00aed\" (UID: \"362d815c-c6ec-48b0-9891-85d06ad00aed\") " Oct 11 10:55:49.264549 master-1 kubenswrapper[4771]: I1011 10:55:49.264499 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-httpd-config" (OuterVolumeSpecName: "httpd-config") pod "362d815c-c6ec-48b0-9891-85d06ad00aed" (UID: "362d815c-c6ec-48b0-9891-85d06ad00aed"). InnerVolumeSpecName "httpd-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:49.265160 master-1 kubenswrapper[4771]: I1011 10:55:49.265117 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362d815c-c6ec-48b0-9891-85d06ad00aed-kube-api-access-n4zt7" (OuterVolumeSpecName: "kube-api-access-n4zt7") pod "362d815c-c6ec-48b0-9891-85d06ad00aed" (UID: "362d815c-c6ec-48b0-9891-85d06ad00aed"). InnerVolumeSpecName "kube-api-access-n4zt7". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:49.298948 master-1 kubenswrapper[4771]: I1011 10:55:49.298846 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:49.298948 master-1 kubenswrapper[4771]: I1011 10:55:49.298958 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:49.305432 master-1 kubenswrapper[4771]: I1011 10:55:49.305333 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "362d815c-c6ec-48b0-9891-85d06ad00aed" (UID: "362d815c-c6ec-48b0-9891-85d06ad00aed"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:49.306145 master-1 kubenswrapper[4771]: I1011 10:55:49.306066 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-config" (OuterVolumeSpecName: "config") pod "362d815c-c6ec-48b0-9891-85d06ad00aed" (UID: "362d815c-c6ec-48b0-9891-85d06ad00aed"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:49.324301 master-1 kubenswrapper[4771]: I1011 10:55:49.324241 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-ovndb-tls-certs" (OuterVolumeSpecName: "ovndb-tls-certs") pod "362d815c-c6ec-48b0-9891-85d06ad00aed" (UID: "362d815c-c6ec-48b0-9891-85d06ad00aed"). InnerVolumeSpecName "ovndb-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:49.338841 master-1 kubenswrapper[4771]: I1011 10:55:49.338761 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-7887b79bcd-stzg5" event={"ID":"362d815c-c6ec-48b0-9891-85d06ad00aed","Type":"ContainerDied","Data":"f93602e6ba46cd0010c3c32ac26a6e16a985dc38b7a78af0515d53b800f6c9e5"} Oct 11 10:55:49.339658 master-1 kubenswrapper[4771]: I1011 10:55:49.338877 4771 scope.go:117] "RemoveContainer" containerID="99502f3eb6699cc67bcf11374ee8446bc01a1a157ce8024301c91ebed596f3f2" Oct 11 10:55:49.340536 master-1 kubenswrapper[4771]: I1011 10:55:49.340496 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-7887b79bcd-stzg5" Oct 11 10:55:49.363661 master-1 kubenswrapper[4771]: I1011 10:55:49.363593 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-n4zt7\" (UniqueName: \"kubernetes.io/projected/362d815c-c6ec-48b0-9891-85d06ad00aed-kube-api-access-n4zt7\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:49.364043 master-1 kubenswrapper[4771]: I1011 10:55:49.364030 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:49.364161 master-1 kubenswrapper[4771]: I1011 10:55:49.364148 4771 reconciler_common.go:293] "Volume detached for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-httpd-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:49.364257 master-1 kubenswrapper[4771]: I1011 10:55:49.364245 4771 reconciler_common.go:293] "Volume detached for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-ovndb-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:49.364340 master-1 kubenswrapper[4771]: I1011 10:55:49.364329 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/362d815c-c6ec-48b0-9891-85d06ad00aed-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:49.365552 master-1 kubenswrapper[4771]: I1011 10:55:49.364072 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:49.404506 master-1 kubenswrapper[4771]: I1011 10:55:49.404445 4771 scope.go:117] "RemoveContainer" containerID="b94dfe1997cbb3d378d19012a9b6401bc1cef35489c7ea7be575908bfe56b3a0" Oct 11 10:55:49.422333 master-1 kubenswrapper[4771]: I1011 10:55:49.422265 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:49.481936 master-1 kubenswrapper[4771]: I1011 10:55:49.481824 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell0-conductor-db-sync-chcrd"] Oct 11 10:55:49.487557 master-1 kubenswrapper[4771]: W1011 10:55:49.487495 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5 WatchSource:0}: Error finding container 0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5: Status 404 returned error can't find the container with id 0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5 Oct 11 10:55:49.741927 master-1 kubenswrapper[4771]: I1011 10:55:49.741846 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/neutron-7887b79bcd-stzg5"] Oct 11 10:55:49.762970 master-1 kubenswrapper[4771]: I1011 10:55:49.760604 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/neutron-7887b79bcd-stzg5"] Oct 11 10:55:49.810161 master-1 kubenswrapper[4771]: I1011 10:55:49.810082 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-a338-account-create-v4xzh" Oct 11 10:55:49.884447 master-1 kubenswrapper[4771]: I1011 10:55:49.884346 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nlsv\" (UniqueName: \"kubernetes.io/projected/7ac9af7f-afc6-4d4d-9923-db14ac820459-kube-api-access-8nlsv\") pod \"7ac9af7f-afc6-4d4d-9923-db14ac820459\" (UID: \"7ac9af7f-afc6-4d4d-9923-db14ac820459\") " Oct 11 10:55:49.888064 master-1 kubenswrapper[4771]: I1011 10:55:49.887980 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac9af7f-afc6-4d4d-9923-db14ac820459-kube-api-access-8nlsv" (OuterVolumeSpecName: "kube-api-access-8nlsv") pod "7ac9af7f-afc6-4d4d-9923-db14ac820459" (UID: "7ac9af7f-afc6-4d4d-9923-db14ac820459"). InnerVolumeSpecName "kube-api-access-8nlsv". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:49.994102 master-1 kubenswrapper[4771]: I1011 10:55:49.987991 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-8nlsv\" (UniqueName: \"kubernetes.io/projected/7ac9af7f-afc6-4d4d-9923-db14ac820459-kube-api-access-8nlsv\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:50.123087 master-1 kubenswrapper[4771]: I1011 10:55:50.123011 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-8vzsw"] Oct 11 10:55:50.351708 master-1 kubenswrapper[4771]: I1011 10:55:50.351631 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerStarted","Data":"b6b58d0d02154c8b742871143df315c38b23b17f3fced652bf59c6663b8ca178"} Oct 11 10:55:50.352562 master-1 kubenswrapper[4771]: I1011 10:55:50.352281 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 10:55:50.353922 master-1 kubenswrapper[4771]: I1011 10:55:50.353884 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-conductor-db-sync-chcrd" event={"ID":"38267a66-0ebd-44ab-bc7f-cd5703503b74","Type":"ContainerStarted","Data":"0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5"} Oct 11 10:55:50.359287 master-1 kubenswrapper[4771]: I1011 10:55:50.359249 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-a338-account-create-v4xzh" Oct 11 10:55:50.359588 master-1 kubenswrapper[4771]: I1011 10:55:50.359555 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-a338-account-create-v4xzh" event={"ID":"7ac9af7f-afc6-4d4d-9923-db14ac820459","Type":"ContainerDied","Data":"9b12e8787780eb4c0e7ae7ec5652e12847e4d2cd1f0a158946584ba9e587475f"} Oct 11 10:55:50.359674 master-1 kubenswrapper[4771]: I1011 10:55:50.359597 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b12e8787780eb4c0e7ae7ec5652e12847e4d2cd1f0a158946584ba9e587475f" Oct 11 10:55:50.461299 master-1 kubenswrapper[4771]: I1011 10:55:50.461122 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="362d815c-c6ec-48b0-9891-85d06ad00aed" path="/var/lib/kubelet/pods/362d815c-c6ec-48b0-9891-85d06ad00aed/volumes" Oct 11 10:55:50.605800 master-1 kubenswrapper[4771]: I1011 10:55:50.605672 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=1.934589968 podStartE2EDuration="6.605637139s" podCreationTimestamp="2025-10-11 10:55:44 +0000 UTC" firstStartedPulling="2025-10-11 10:55:45.211733768 +0000 UTC m=+1777.185960209" lastFinishedPulling="2025-10-11 10:55:49.882780939 +0000 UTC m=+1781.857007380" observedRunningTime="2025-10-11 10:55:50.598621075 +0000 UTC m=+1782.572847526" watchObservedRunningTime="2025-10-11 10:55:50.605637139 +0000 UTC m=+1782.579863610" Oct 11 10:55:51.369655 master-1 kubenswrapper[4771]: I1011 10:55:51.369579 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-operators-8vzsw" podUID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerName="registry-server" containerID="cri-o://00b0738c103a1f8936204d6f12df3d6fd67321868af021b54204d55c141f77ca" gracePeriod=2 Oct 11 10:55:51.701839 master-1 kubenswrapper[4771]: I1011 10:55:51.701649 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ironic-7cddc977f5-9ddgm"] Oct 11 10:55:51.702083 master-1 kubenswrapper[4771]: I1011 10:55:51.702041 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ironic-7cddc977f5-9ddgm" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api-log" containerID="cri-o://a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e" gracePeriod=60 Oct 11 10:55:52.318587 master-1 kubenswrapper[4771]: I1011 10:55:52.318071 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:52.344879 master-1 kubenswrapper[4771]: I1011 10:55:52.344782 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-scripts\") pod \"879970ca-6312-4aec-b8f4-a8a41a0e3797\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " Oct 11 10:55:52.345230 master-1 kubenswrapper[4771]: I1011 10:55:52.344913 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4pzb\" (UniqueName: \"kubernetes.io/projected/879970ca-6312-4aec-b8f4-a8a41a0e3797-kube-api-access-k4pzb\") pod \"879970ca-6312-4aec-b8f4-a8a41a0e3797\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " Oct 11 10:55:52.345230 master-1 kubenswrapper[4771]: I1011 10:55:52.344989 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data\") pod \"879970ca-6312-4aec-b8f4-a8a41a0e3797\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " Oct 11 10:55:52.345230 master-1 kubenswrapper[4771]: I1011 10:55:52.345037 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-custom\") pod \"879970ca-6312-4aec-b8f4-a8a41a0e3797\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " Oct 11 10:55:52.345230 master-1 kubenswrapper[4771]: I1011 10:55:52.345066 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-logs\") pod \"879970ca-6312-4aec-b8f4-a8a41a0e3797\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " Oct 11 10:55:52.345230 master-1 kubenswrapper[4771]: I1011 10:55:52.345095 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-combined-ca-bundle\") pod \"879970ca-6312-4aec-b8f4-a8a41a0e3797\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " Oct 11 10:55:52.345230 master-1 kubenswrapper[4771]: I1011 10:55:52.345235 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-merged\") pod \"879970ca-6312-4aec-b8f4-a8a41a0e3797\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " Oct 11 10:55:52.345506 master-1 kubenswrapper[4771]: I1011 10:55:52.345259 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etc-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879970ca-6312-4aec-b8f4-a8a41a0e3797-etc-podinfo\") pod \"879970ca-6312-4aec-b8f4-a8a41a0e3797\" (UID: \"879970ca-6312-4aec-b8f4-a8a41a0e3797\") " Oct 11 10:55:52.349682 master-1 kubenswrapper[4771]: I1011 10:55:52.349562 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-custom" (OuterVolumeSpecName: "config-data-custom") pod "879970ca-6312-4aec-b8f4-a8a41a0e3797" (UID: "879970ca-6312-4aec-b8f4-a8a41a0e3797"). InnerVolumeSpecName "config-data-custom". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:52.349682 master-1 kubenswrapper[4771]: I1011 10:55:52.349603 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/879970ca-6312-4aec-b8f4-a8a41a0e3797-etc-podinfo" (OuterVolumeSpecName: "etc-podinfo") pod "879970ca-6312-4aec-b8f4-a8a41a0e3797" (UID: "879970ca-6312-4aec-b8f4-a8a41a0e3797"). InnerVolumeSpecName "etc-podinfo". PluginName "kubernetes.io/downward-api", VolumeGidValue "" Oct 11 10:55:52.349682 master-1 kubenswrapper[4771]: I1011 10:55:52.349575 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879970ca-6312-4aec-b8f4-a8a41a0e3797-kube-api-access-k4pzb" (OuterVolumeSpecName: "kube-api-access-k4pzb") pod "879970ca-6312-4aec-b8f4-a8a41a0e3797" (UID: "879970ca-6312-4aec-b8f4-a8a41a0e3797"). InnerVolumeSpecName "kube-api-access-k4pzb". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:52.349968 master-1 kubenswrapper[4771]: I1011 10:55:52.349882 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-logs" (OuterVolumeSpecName: "logs") pod "879970ca-6312-4aec-b8f4-a8a41a0e3797" (UID: "879970ca-6312-4aec-b8f4-a8a41a0e3797"). InnerVolumeSpecName "logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:52.350578 master-1 kubenswrapper[4771]: I1011 10:55:52.350529 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-scripts" (OuterVolumeSpecName: "scripts") pod "879970ca-6312-4aec-b8f4-a8a41a0e3797" (UID: "879970ca-6312-4aec-b8f4-a8a41a0e3797"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:52.358564 master-1 kubenswrapper[4771]: I1011 10:55:52.358380 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-merged" (OuterVolumeSpecName: "config-data-merged") pod "879970ca-6312-4aec-b8f4-a8a41a0e3797" (UID: "879970ca-6312-4aec-b8f4-a8a41a0e3797"). InnerVolumeSpecName "config-data-merged". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:52.368843 master-1 kubenswrapper[4771]: I1011 10:55:52.368763 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data" (OuterVolumeSpecName: "config-data") pod "879970ca-6312-4aec-b8f4-a8a41a0e3797" (UID: "879970ca-6312-4aec-b8f4-a8a41a0e3797"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:52.386196 master-1 kubenswrapper[4771]: I1011 10:55:52.386104 4771 generic.go:334] "Generic (PLEG): container finished" podID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerID="a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e" exitCode=143 Oct 11 10:55:52.386739 master-1 kubenswrapper[4771]: I1011 10:55:52.386237 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-7cddc977f5-9ddgm" event={"ID":"879970ca-6312-4aec-b8f4-a8a41a0e3797","Type":"ContainerDied","Data":"a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e"} Oct 11 10:55:52.386739 master-1 kubenswrapper[4771]: I1011 10:55:52.386279 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-7cddc977f5-9ddgm" event={"ID":"879970ca-6312-4aec-b8f4-a8a41a0e3797","Type":"ContainerDied","Data":"3182b36cd0161342e4b24feff5f2f372022de500b18e279f9cd6a7f20bca373c"} Oct 11 10:55:52.386739 master-1 kubenswrapper[4771]: I1011 10:55:52.386308 4771 scope.go:117] "RemoveContainer" containerID="d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb" Oct 11 10:55:52.387348 master-1 kubenswrapper[4771]: I1011 10:55:52.386835 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ironic-7cddc977f5-9ddgm" Oct 11 10:55:52.398547 master-1 kubenswrapper[4771]: I1011 10:55:52.398464 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "879970ca-6312-4aec-b8f4-a8a41a0e3797" (UID: "879970ca-6312-4aec-b8f4-a8a41a0e3797"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:52.410636 master-1 kubenswrapper[4771]: I1011 10:55:52.410440 4771 generic.go:334] "Generic (PLEG): container finished" podID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerID="00b0738c103a1f8936204d6f12df3d6fd67321868af021b54204d55c141f77ca" exitCode=0 Oct 11 10:55:52.410636 master-1 kubenswrapper[4771]: I1011 10:55:52.410515 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-8vzsw" event={"ID":"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44","Type":"ContainerDied","Data":"00b0738c103a1f8936204d6f12df3d6fd67321868af021b54204d55c141f77ca"} Oct 11 10:55:52.422209 master-1 kubenswrapper[4771]: I1011 10:55:52.421998 4771 scope.go:117] "RemoveContainer" containerID="a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e" Oct 11 10:55:52.442764 master-1 kubenswrapper[4771]: I1011 10:55:52.442718 4771 scope.go:117] "RemoveContainer" containerID="ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104" Oct 11 10:55:52.447507 master-1 kubenswrapper[4771]: I1011 10:55:52.447454 4771 reconciler_common.go:293] "Volume detached for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-custom\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.447507 master-1 kubenswrapper[4771]: I1011 10:55:52.447502 4771 reconciler_common.go:293] "Volume detached for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-logs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.448181 master-1 kubenswrapper[4771]: I1011 10:55:52.447518 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.448181 master-1 kubenswrapper[4771]: I1011 10:55:52.447534 4771 reconciler_common.go:293] "Volume detached for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data-merged\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.448181 master-1 kubenswrapper[4771]: I1011 10:55:52.447551 4771 reconciler_common.go:293] "Volume detached for volume \"etc-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879970ca-6312-4aec-b8f4-a8a41a0e3797-etc-podinfo\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.448181 master-1 kubenswrapper[4771]: I1011 10:55:52.447564 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.448181 master-1 kubenswrapper[4771]: I1011 10:55:52.447578 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-k4pzb\" (UniqueName: \"kubernetes.io/projected/879970ca-6312-4aec-b8f4-a8a41a0e3797-kube-api-access-k4pzb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.448181 master-1 kubenswrapper[4771]: I1011 10:55:52.447590 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/879970ca-6312-4aec-b8f4-a8a41a0e3797-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.459154 master-1 kubenswrapper[4771]: I1011 10:55:52.459101 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:52.465031 master-1 kubenswrapper[4771]: I1011 10:55:52.464984 4771 scope.go:117] "RemoveContainer" containerID="d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb" Oct 11 10:55:52.465460 master-1 kubenswrapper[4771]: E1011 10:55:52.465412 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb\": container with ID starting with d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb not found: ID does not exist" containerID="d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb" Oct 11 10:55:52.465529 master-1 kubenswrapper[4771]: I1011 10:55:52.465465 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb"} err="failed to get container status \"d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb\": rpc error: code = NotFound desc = could not find container \"d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb\": container with ID starting with d960a6ebc10578bac963a4210bd8cc373ed3887cebb614d8345d375a88c7faeb not found: ID does not exist" Oct 11 10:55:52.465529 master-1 kubenswrapper[4771]: I1011 10:55:52.465503 4771 scope.go:117] "RemoveContainer" containerID="a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e" Oct 11 10:55:52.465894 master-1 kubenswrapper[4771]: E1011 10:55:52.465860 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e\": container with ID starting with a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e not found: ID does not exist" containerID="a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e" Oct 11 10:55:52.465939 master-1 kubenswrapper[4771]: I1011 10:55:52.465891 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e"} err="failed to get container status \"a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e\": rpc error: code = NotFound desc = could not find container \"a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e\": container with ID starting with a3bee7ce4b0c367a06f8dac0455b4363a2d71194a9474e87524e9ae41ec4264e not found: ID does not exist" Oct 11 10:55:52.465939 master-1 kubenswrapper[4771]: I1011 10:55:52.465909 4771 scope.go:117] "RemoveContainer" containerID="ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104" Oct 11 10:55:52.466219 master-1 kubenswrapper[4771]: E1011 10:55:52.466186 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104\": container with ID starting with ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104 not found: ID does not exist" containerID="ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104" Oct 11 10:55:52.466276 master-1 kubenswrapper[4771]: I1011 10:55:52.466232 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104"} err="failed to get container status \"ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104\": rpc error: code = NotFound desc = could not find container \"ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104\": container with ID starting with ef95a48f33dffbcc5025cb68d51739a664704d5571f4c1f1a0bf787ab1898104 not found: ID does not exist" Oct 11 10:55:52.548950 master-1 kubenswrapper[4771]: I1011 10:55:52.548768 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sl2p\" (UniqueName: \"kubernetes.io/projected/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-kube-api-access-8sl2p\") pod \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " Oct 11 10:55:52.548950 master-1 kubenswrapper[4771]: I1011 10:55:52.548866 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-catalog-content\") pod \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " Oct 11 10:55:52.549501 master-1 kubenswrapper[4771]: I1011 10:55:52.548961 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-utilities\") pod \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\" (UID: \"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44\") " Oct 11 10:55:52.550310 master-1 kubenswrapper[4771]: I1011 10:55:52.550265 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-utilities" (OuterVolumeSpecName: "utilities") pod "1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" (UID: "1daf9046-c7b7-4c9c-a9b3-76ae17e47e44"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:52.553500 master-1 kubenswrapper[4771]: I1011 10:55:52.553428 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-kube-api-access-8sl2p" (OuterVolumeSpecName: "kube-api-access-8sl2p") pod "1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" (UID: "1daf9046-c7b7-4c9c-a9b3-76ae17e47e44"). InnerVolumeSpecName "kube-api-access-8sl2p". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:52.659392 master-1 kubenswrapper[4771]: I1011 10:55:52.659291 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" (UID: "1daf9046-c7b7-4c9c-a9b3-76ae17e47e44"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:52.665827 master-1 kubenswrapper[4771]: I1011 10:55:52.665770 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.666156 master-1 kubenswrapper[4771]: I1011 10:55:52.666142 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-8sl2p\" (UniqueName: \"kubernetes.io/projected/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-kube-api-access-8sl2p\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:52.666235 master-1 kubenswrapper[4771]: I1011 10:55:52.666222 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:53.263379 master-1 kubenswrapper[4771]: I1011 10:55:53.263152 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ironic-7cddc977f5-9ddgm"] Oct 11 10:55:53.423193 master-1 kubenswrapper[4771]: I1011 10:55:53.423110 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-8vzsw" event={"ID":"1daf9046-c7b7-4c9c-a9b3-76ae17e47e44","Type":"ContainerDied","Data":"2e03e915f6f95ecc8f0f52052466e21bd1b0bb1a12eb203399bd0345ac65bccf"} Oct 11 10:55:53.423193 master-1 kubenswrapper[4771]: I1011 10:55:53.423197 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-8vzsw" Oct 11 10:55:53.424006 master-1 kubenswrapper[4771]: I1011 10:55:53.423211 4771 scope.go:117] "RemoveContainer" containerID="00b0738c103a1f8936204d6f12df3d6fd67321868af021b54204d55c141f77ca" Oct 11 10:55:53.511778 master-1 kubenswrapper[4771]: E1011 10:55:53.511710 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daf9046_c7b7_4c9c_a9b3_76ae17e47e44.slice/crio-2e03e915f6f95ecc8f0f52052466e21bd1b0bb1a12eb203399bd0345ac65bccf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daf9046_c7b7_4c9c_a9b3_76ae17e47e44.slice\": RecentStats: unable to find data in memory cache]" Oct 11 10:55:53.512292 master-1 kubenswrapper[4771]: E1011 10:55:53.512248 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daf9046_c7b7_4c9c_a9b3_76ae17e47e44.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daf9046_c7b7_4c9c_a9b3_76ae17e47e44.slice/crio-2e03e915f6f95ecc8f0f52052466e21bd1b0bb1a12eb203399bd0345ac65bccf\": RecentStats: unable to find data in memory cache]" Oct 11 10:55:53.638783 master-1 kubenswrapper[4771]: I1011 10:55:53.638695 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/community-operators-4bbqs"] Oct 11 10:55:53.639131 master-1 kubenswrapper[4771]: E1011 10:55:53.639110 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerName="extract-utilities" Oct 11 10:55:53.639131 master-1 kubenswrapper[4771]: I1011 10:55:53.639125 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerName="extract-utilities" Oct 11 10:55:53.639212 master-1 kubenswrapper[4771]: E1011 10:55:53.639152 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerName="registry-server" Oct 11 10:55:53.639212 master-1 kubenswrapper[4771]: I1011 10:55:53.639159 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerName="registry-server" Oct 11 10:55:53.639212 master-1 kubenswrapper[4771]: E1011 10:55:53.639173 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerName="extract-content" Oct 11 10:55:53.639212 master-1 kubenswrapper[4771]: I1011 10:55:53.639180 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerName="extract-content" Oct 11 10:55:53.639212 master-1 kubenswrapper[4771]: E1011 10:55:53.639192 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api" Oct 11 10:55:53.639212 master-1 kubenswrapper[4771]: I1011 10:55:53.639199 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api" Oct 11 10:55:53.639212 master-1 kubenswrapper[4771]: E1011 10:55:53.639215 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="init" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: I1011 10:55:53.639222 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="init" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: E1011 10:55:53.639236 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerName="neutron-httpd" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: I1011 10:55:53.639244 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerName="neutron-httpd" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: E1011 10:55:53.639254 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="7ac9af7f-afc6-4d4d-9923-db14ac820459" containerName="mariadb-account-create" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: I1011 10:55:53.639259 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac9af7f-afc6-4d4d-9923-db14ac820459" containerName="mariadb-account-create" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: E1011 10:55:53.639268 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerName="neutron-api" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: I1011 10:55:53.639273 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerName="neutron-api" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: E1011 10:55:53.639284 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api-log" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: I1011 10:55:53.639290 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api-log" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: I1011 10:55:53.639440 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: I1011 10:55:53.639456 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerName="neutron-api" Oct 11 10:55:53.639491 master-1 kubenswrapper[4771]: I1011 10:55:53.639466 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api" Oct 11 10:55:53.655446 master-1 kubenswrapper[4771]: I1011 10:55:53.639978 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" containerName="registry-server" Oct 11 10:55:53.655446 master-1 kubenswrapper[4771]: I1011 10:55:53.639990 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="362d815c-c6ec-48b0-9891-85d06ad00aed" containerName="neutron-httpd" Oct 11 10:55:53.655446 master-1 kubenswrapper[4771]: I1011 10:55:53.640003 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api-log" Oct 11 10:55:53.655446 master-1 kubenswrapper[4771]: I1011 10:55:53.640010 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="7ac9af7f-afc6-4d4d-9923-db14ac820459" containerName="mariadb-account-create" Oct 11 10:55:53.655446 master-1 kubenswrapper[4771]: E1011 10:55:53.640241 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api" Oct 11 10:55:53.655446 master-1 kubenswrapper[4771]: I1011 10:55:53.640250 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" containerName="ironic-api" Oct 11 10:55:53.655446 master-1 kubenswrapper[4771]: I1011 10:55:53.642245 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:53.689317 master-1 kubenswrapper[4771]: I1011 10:55:53.689250 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzpqz\" (UniqueName: \"kubernetes.io/projected/5921e565-c581-42f4-8da8-df72fae9a3c0-kube-api-access-jzpqz\") pod \"community-operators-4bbqs\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:53.689694 master-1 kubenswrapper[4771]: I1011 10:55:53.689450 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-utilities\") pod \"community-operators-4bbqs\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:53.689897 master-1 kubenswrapper[4771]: I1011 10:55:53.689852 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-catalog-content\") pod \"community-operators-4bbqs\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:53.793673 master-1 kubenswrapper[4771]: I1011 10:55:53.793581 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-catalog-content\") pod \"community-operators-4bbqs\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:53.794079 master-1 kubenswrapper[4771]: I1011 10:55:53.793784 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-jzpqz\" (UniqueName: \"kubernetes.io/projected/5921e565-c581-42f4-8da8-df72fae9a3c0-kube-api-access-jzpqz\") pod \"community-operators-4bbqs\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:53.794079 master-1 kubenswrapper[4771]: I1011 10:55:53.793811 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-utilities\") pod \"community-operators-4bbqs\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:53.794241 master-1 kubenswrapper[4771]: I1011 10:55:53.794216 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-catalog-content\") pod \"community-operators-4bbqs\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:53.794241 master-1 kubenswrapper[4771]: I1011 10:55:53.794235 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-utilities\") pod \"community-operators-4bbqs\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:53.978597 master-1 kubenswrapper[4771]: I1011 10:55:53.978435 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-4bbqs"] Oct 11 10:55:53.998545 master-1 kubenswrapper[4771]: I1011 10:55:53.998464 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ironic-7cddc977f5-9ddgm"] Oct 11 10:55:54.230860 master-1 kubenswrapper[4771]: I1011 10:55:54.228815 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-8vzsw"] Oct 11 10:55:54.246496 master-1 kubenswrapper[4771]: I1011 10:55:54.241873 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzpqz\" (UniqueName: \"kubernetes.io/projected/5921e565-c581-42f4-8da8-df72fae9a3c0-kube-api-access-jzpqz\") pod \"community-operators-4bbqs\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:54.274793 master-1 kubenswrapper[4771]: I1011 10:55:54.274738 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:55:54.300684 master-1 kubenswrapper[4771]: I1011 10:55:54.299108 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-operators-8vzsw"] Oct 11 10:55:54.453461 master-1 kubenswrapper[4771]: I1011 10:55:54.453379 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1daf9046-c7b7-4c9c-a9b3-76ae17e47e44" path="/var/lib/kubelet/pods/1daf9046-c7b7-4c9c-a9b3-76ae17e47e44/volumes" Oct 11 10:55:54.454438 master-1 kubenswrapper[4771]: I1011 10:55:54.454416 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="879970ca-6312-4aec-b8f4-a8a41a0e3797" path="/var/lib/kubelet/pods/879970ca-6312-4aec-b8f4-a8a41a0e3797/volumes" Oct 11 10:55:56.636909 master-1 kubenswrapper[4771]: I1011 10:55:56.636801 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:56.638131 master-1 kubenswrapper[4771]: I1011 10:55:56.637339 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="ceilometer-central-agent" containerID="cri-o://927b471b1fa44ac0453124772c1c315e44a9bd395c3c5d07c01293a49d70a8ef" gracePeriod=30 Oct 11 10:55:56.638131 master-1 kubenswrapper[4771]: I1011 10:55:56.637544 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="proxy-httpd" containerID="cri-o://b6b58d0d02154c8b742871143df315c38b23b17f3fced652bf59c6663b8ca178" gracePeriod=30 Oct 11 10:55:56.638131 master-1 kubenswrapper[4771]: I1011 10:55:56.637588 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="sg-core" containerID="cri-o://dea9e948550286a77692eb6be22395f88e3b28cb639cfe7f80a137916ddb5ac1" gracePeriod=30 Oct 11 10:55:56.638131 master-1 kubenswrapper[4771]: I1011 10:55:56.637920 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="ceilometer-notification-agent" containerID="cri-o://e699798eb3d463b27b94ec4cd6a4bacb981768b5034628ba7f0e14e667c53445" gracePeriod=30 Oct 11 10:55:57.466677 master-1 kubenswrapper[4771]: I1011 10:55:57.466510 4771 generic.go:334] "Generic (PLEG): container finished" podID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerID="b6b58d0d02154c8b742871143df315c38b23b17f3fced652bf59c6663b8ca178" exitCode=0 Oct 11 10:55:57.466677 master-1 kubenswrapper[4771]: I1011 10:55:57.466571 4771 generic.go:334] "Generic (PLEG): container finished" podID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerID="dea9e948550286a77692eb6be22395f88e3b28cb639cfe7f80a137916ddb5ac1" exitCode=2 Oct 11 10:55:57.466677 master-1 kubenswrapper[4771]: I1011 10:55:57.466580 4771 generic.go:334] "Generic (PLEG): container finished" podID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerID="e699798eb3d463b27b94ec4cd6a4bacb981768b5034628ba7f0e14e667c53445" exitCode=0 Oct 11 10:55:57.466677 master-1 kubenswrapper[4771]: I1011 10:55:57.466589 4771 generic.go:334] "Generic (PLEG): container finished" podID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerID="927b471b1fa44ac0453124772c1c315e44a9bd395c3c5d07c01293a49d70a8ef" exitCode=0 Oct 11 10:55:57.466677 master-1 kubenswrapper[4771]: I1011 10:55:57.466628 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerDied","Data":"b6b58d0d02154c8b742871143df315c38b23b17f3fced652bf59c6663b8ca178"} Oct 11 10:55:57.466677 master-1 kubenswrapper[4771]: I1011 10:55:57.466657 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerDied","Data":"dea9e948550286a77692eb6be22395f88e3b28cb639cfe7f80a137916ddb5ac1"} Oct 11 10:55:57.466677 master-1 kubenswrapper[4771]: I1011 10:55:57.466668 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerDied","Data":"e699798eb3d463b27b94ec4cd6a4bacb981768b5034628ba7f0e14e667c53445"} Oct 11 10:55:57.467158 master-1 kubenswrapper[4771]: I1011 10:55:57.466700 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerDied","Data":"927b471b1fa44ac0453124772c1c315e44a9bd395c3c5d07c01293a49d70a8ef"} Oct 11 10:55:58.235210 master-1 kubenswrapper[4771]: I1011 10:55:58.235133 4771 scope.go:117] "RemoveContainer" containerID="f4caf5c874767c7fd75c2e84ff37e1b5c988f50fb776ae2062994f2e951ecc23" Oct 11 10:55:58.377775 master-1 kubenswrapper[4771]: I1011 10:55:58.376597 4771 scope.go:117] "RemoveContainer" containerID="e046736cf54a6f375a2d21055bc37323ff6d218a499c8b0059aa035f5e4d1a0c" Oct 11 10:55:58.659418 master-1 kubenswrapper[4771]: I1011 10:55:58.657481 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:58.734731 master-1 kubenswrapper[4771]: I1011 10:55:58.734669 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-scripts\") pod \"0f6abca2-1aea-4da7-88aa-1d7651959165\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " Oct 11 10:55:58.734966 master-1 kubenswrapper[4771]: I1011 10:55:58.734951 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-sg-core-conf-yaml\") pod \"0f6abca2-1aea-4da7-88aa-1d7651959165\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " Oct 11 10:55:58.735795 master-1 kubenswrapper[4771]: I1011 10:55:58.735761 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-config-data\") pod \"0f6abca2-1aea-4da7-88aa-1d7651959165\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " Oct 11 10:55:58.735844 master-1 kubenswrapper[4771]: I1011 10:55:58.735829 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl7xd\" (UniqueName: \"kubernetes.io/projected/0f6abca2-1aea-4da7-88aa-1d7651959165-kube-api-access-pl7xd\") pod \"0f6abca2-1aea-4da7-88aa-1d7651959165\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " Oct 11 10:55:58.736947 master-1 kubenswrapper[4771]: I1011 10:55:58.736919 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "0f6abca2-1aea-4da7-88aa-1d7651959165" (UID: "0f6abca2-1aea-4da7-88aa-1d7651959165"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:58.737048 master-1 kubenswrapper[4771]: I1011 10:55:58.736960 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-log-httpd\") pod \"0f6abca2-1aea-4da7-88aa-1d7651959165\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " Oct 11 10:55:58.737048 master-1 kubenswrapper[4771]: I1011 10:55:58.737017 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-run-httpd\") pod \"0f6abca2-1aea-4da7-88aa-1d7651959165\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " Oct 11 10:55:58.738107 master-1 kubenswrapper[4771]: I1011 10:55:58.738083 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "0f6abca2-1aea-4da7-88aa-1d7651959165" (UID: "0f6abca2-1aea-4da7-88aa-1d7651959165"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:55:58.739461 master-1 kubenswrapper[4771]: I1011 10:55:58.739380 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-scripts" (OuterVolumeSpecName: "scripts") pod "0f6abca2-1aea-4da7-88aa-1d7651959165" (UID: "0f6abca2-1aea-4da7-88aa-1d7651959165"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:58.739645 master-1 kubenswrapper[4771]: I1011 10:55:58.737177 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-combined-ca-bundle\") pod \"0f6abca2-1aea-4da7-88aa-1d7651959165\" (UID: \"0f6abca2-1aea-4da7-88aa-1d7651959165\") " Oct 11 10:55:58.740136 master-1 kubenswrapper[4771]: I1011 10:55:58.740094 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6abca2-1aea-4da7-88aa-1d7651959165-kube-api-access-pl7xd" (OuterVolumeSpecName: "kube-api-access-pl7xd") pod "0f6abca2-1aea-4da7-88aa-1d7651959165" (UID: "0f6abca2-1aea-4da7-88aa-1d7651959165"). InnerVolumeSpecName "kube-api-access-pl7xd". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:55:58.743767 master-1 kubenswrapper[4771]: I1011 10:55:58.743700 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:58.743867 master-1 kubenswrapper[4771]: I1011 10:55:58.743794 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-pl7xd\" (UniqueName: \"kubernetes.io/projected/0f6abca2-1aea-4da7-88aa-1d7651959165-kube-api-access-pl7xd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:58.743867 master-1 kubenswrapper[4771]: I1011 10:55:58.743816 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:58.743867 master-1 kubenswrapper[4771]: I1011 10:55:58.743830 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/0f6abca2-1aea-4da7-88aa-1d7651959165-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:58.763202 master-1 kubenswrapper[4771]: I1011 10:55:58.763096 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "0f6abca2-1aea-4da7-88aa-1d7651959165" (UID: "0f6abca2-1aea-4da7-88aa-1d7651959165"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:58.783863 master-1 kubenswrapper[4771]: I1011 10:55:58.783120 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-4bbqs"] Oct 11 10:55:58.792402 master-1 kubenswrapper[4771]: W1011 10:55:58.792326 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe WatchSource:0}: Error finding container 2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe: Status 404 returned error can't find the container with id 2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe Oct 11 10:55:58.822518 master-1 kubenswrapper[4771]: I1011 10:55:58.822461 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "0f6abca2-1aea-4da7-88aa-1d7651959165" (UID: "0f6abca2-1aea-4da7-88aa-1d7651959165"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:58.828803 master-1 kubenswrapper[4771]: I1011 10:55:58.828732 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-config-data" (OuterVolumeSpecName: "config-data") pod "0f6abca2-1aea-4da7-88aa-1d7651959165" (UID: "0f6abca2-1aea-4da7-88aa-1d7651959165"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:55:58.846643 master-1 kubenswrapper[4771]: I1011 10:55:58.846592 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:58.846755 master-1 kubenswrapper[4771]: I1011 10:55:58.846650 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:58.846755 master-1 kubenswrapper[4771]: I1011 10:55:58.846665 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/0f6abca2-1aea-4da7-88aa-1d7651959165-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 10:55:59.048635 master-1 kubenswrapper[4771]: E1011 10:55:59.048011 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-conmon-57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:55:59.376883 master-1 kubenswrapper[4771]: I1011 10:55:59.350499 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:55:59.376883 master-1 kubenswrapper[4771]: I1011 10:55:59.350838 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/glance-b5802-default-internal-api-1" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-log" containerID="cri-o://9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e" gracePeriod=30 Oct 11 10:55:59.376883 master-1 kubenswrapper[4771]: I1011 10:55:59.350968 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/glance-b5802-default-internal-api-1" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-httpd" containerID="cri-o://20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57" gracePeriod=30 Oct 11 10:55:59.505662 master-1 kubenswrapper[4771]: I1011 10:55:59.505108 4771 generic.go:334] "Generic (PLEG): container finished" podID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerID="57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346" exitCode=0 Oct 11 10:55:59.505662 master-1 kubenswrapper[4771]: I1011 10:55:59.505212 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-4bbqs" event={"ID":"5921e565-c581-42f4-8da8-df72fae9a3c0","Type":"ContainerDied","Data":"57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346"} Oct 11 10:55:59.505662 master-1 kubenswrapper[4771]: I1011 10:55:59.505259 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-4bbqs" event={"ID":"5921e565-c581-42f4-8da8-df72fae9a3c0","Type":"ContainerStarted","Data":"2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe"} Oct 11 10:55:59.511109 master-1 kubenswrapper[4771]: I1011 10:55:59.511061 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"0f6abca2-1aea-4da7-88aa-1d7651959165","Type":"ContainerDied","Data":"84ce7b87c3fe4abde506e09dd4965353c30550d9eae5707b3cd6ecad602405a9"} Oct 11 10:55:59.511220 master-1 kubenswrapper[4771]: I1011 10:55:59.511125 4771 scope.go:117] "RemoveContainer" containerID="b6b58d0d02154c8b742871143df315c38b23b17f3fced652bf59c6663b8ca178" Oct 11 10:55:59.511389 master-1 kubenswrapper[4771]: I1011 10:55:59.511312 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:59.526723 master-1 kubenswrapper[4771]: I1011 10:55:59.526668 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-conductor-db-sync-chcrd" event={"ID":"38267a66-0ebd-44ab-bc7f-cd5703503b74","Type":"ContainerStarted","Data":"5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed"} Oct 11 10:55:59.550851 master-1 kubenswrapper[4771]: I1011 10:55:59.550821 4771 scope.go:117] "RemoveContainer" containerID="dea9e948550286a77692eb6be22395f88e3b28cb639cfe7f80a137916ddb5ac1" Oct 11 10:55:59.565712 master-1 kubenswrapper[4771]: I1011 10:55:59.565552 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:59.573573 master-1 kubenswrapper[4771]: I1011 10:55:59.573500 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:59.575442 master-1 kubenswrapper[4771]: I1011 10:55:59.575417 4771 scope.go:117] "RemoveContainer" containerID="e699798eb3d463b27b94ec4cd6a4bacb981768b5034628ba7f0e14e667c53445" Oct 11 10:55:59.593237 master-1 kubenswrapper[4771]: I1011 10:55:59.593108 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell0-conductor-db-sync-chcrd" podStartSLOduration=3.954474251 podStartE2EDuration="12.593083317s" podCreationTimestamp="2025-10-11 10:55:47 +0000 UTC" firstStartedPulling="2025-10-11 10:55:49.632283683 +0000 UTC m=+1781.606510124" lastFinishedPulling="2025-10-11 10:55:58.270892749 +0000 UTC m=+1790.245119190" observedRunningTime="2025-10-11 10:55:59.590065229 +0000 UTC m=+1791.564291680" watchObservedRunningTime="2025-10-11 10:55:59.593083317 +0000 UTC m=+1791.567309768" Oct 11 10:55:59.606772 master-1 kubenswrapper[4771]: I1011 10:55:59.606713 4771 scope.go:117] "RemoveContainer" containerID="927b471b1fa44ac0453124772c1c315e44a9bd395c3c5d07c01293a49d70a8ef" Oct 11 10:55:59.610535 master-1 kubenswrapper[4771]: I1011 10:55:59.610497 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:59.610938 master-1 kubenswrapper[4771]: E1011 10:55:59.610909 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="ceilometer-notification-agent" Oct 11 10:55:59.610938 master-1 kubenswrapper[4771]: I1011 10:55:59.610934 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="ceilometer-notification-agent" Oct 11 10:55:59.611037 master-1 kubenswrapper[4771]: E1011 10:55:59.610988 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="ceilometer-central-agent" Oct 11 10:55:59.611037 master-1 kubenswrapper[4771]: I1011 10:55:59.611000 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="ceilometer-central-agent" Oct 11 10:55:59.611037 master-1 kubenswrapper[4771]: E1011 10:55:59.611028 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="proxy-httpd" Oct 11 10:55:59.611162 master-1 kubenswrapper[4771]: I1011 10:55:59.611039 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="proxy-httpd" Oct 11 10:55:59.611162 master-1 kubenswrapper[4771]: E1011 10:55:59.611068 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="sg-core" Oct 11 10:55:59.611162 master-1 kubenswrapper[4771]: I1011 10:55:59.611078 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="sg-core" Oct 11 10:55:59.611325 master-1 kubenswrapper[4771]: I1011 10:55:59.611300 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="ceilometer-central-agent" Oct 11 10:55:59.611410 master-1 kubenswrapper[4771]: I1011 10:55:59.611342 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="sg-core" Oct 11 10:55:59.611410 master-1 kubenswrapper[4771]: I1011 10:55:59.611374 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="proxy-httpd" Oct 11 10:55:59.611410 master-1 kubenswrapper[4771]: I1011 10:55:59.611393 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" containerName="ceilometer-notification-agent" Oct 11 10:55:59.613382 master-1 kubenswrapper[4771]: I1011 10:55:59.613344 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:55:59.618517 master-1 kubenswrapper[4771]: I1011 10:55:59.618490 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 10:55:59.618592 master-1 kubenswrapper[4771]: I1011 10:55:59.618523 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 10:55:59.628152 master-1 kubenswrapper[4771]: I1011 10:55:59.628093 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:55:59.670698 master-1 kubenswrapper[4771]: I1011 10:55:59.670620 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-config-data\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.671043 master-1 kubenswrapper[4771]: I1011 10:55:59.670741 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.671043 master-1 kubenswrapper[4771]: I1011 10:55:59.670769 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dsl\" (UniqueName: \"kubernetes.io/projected/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-kube-api-access-75dsl\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.671043 master-1 kubenswrapper[4771]: I1011 10:55:59.670797 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-scripts\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.671043 master-1 kubenswrapper[4771]: I1011 10:55:59.670848 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.671236 master-1 kubenswrapper[4771]: I1011 10:55:59.671076 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-run-httpd\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.671341 master-1 kubenswrapper[4771]: I1011 10:55:59.671285 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-log-httpd\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.774844 master-1 kubenswrapper[4771]: I1011 10:55:59.773704 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-scripts\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.774844 master-1 kubenswrapper[4771]: I1011 10:55:59.773819 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.774844 master-1 kubenswrapper[4771]: I1011 10:55:59.773855 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-run-httpd\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.774844 master-1 kubenswrapper[4771]: I1011 10:55:59.773895 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-log-httpd\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.774844 master-1 kubenswrapper[4771]: I1011 10:55:59.773976 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-config-data\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.774844 master-1 kubenswrapper[4771]: I1011 10:55:59.774047 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.774844 master-1 kubenswrapper[4771]: I1011 10:55:59.774075 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-75dsl\" (UniqueName: \"kubernetes.io/projected/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-kube-api-access-75dsl\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.775375 master-1 kubenswrapper[4771]: I1011 10:55:59.774838 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-log-httpd\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.775375 master-1 kubenswrapper[4771]: I1011 10:55:59.775159 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-run-httpd\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.777948 master-1 kubenswrapper[4771]: I1011 10:55:59.777902 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-scripts\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.778342 master-1 kubenswrapper[4771]: I1011 10:55:59.778311 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.779387 master-1 kubenswrapper[4771]: I1011 10:55:59.779348 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-config-data\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.800190 master-1 kubenswrapper[4771]: I1011 10:55:59.800140 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dsl\" (UniqueName: \"kubernetes.io/projected/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-kube-api-access-75dsl\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.806220 master-1 kubenswrapper[4771]: I1011 10:55:59.806183 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " pod="openstack/ceilometer-0" Oct 11 10:55:59.933669 master-1 kubenswrapper[4771]: I1011 10:55:59.933494 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:56:00.409884 master-1 kubenswrapper[4771]: I1011 10:56:00.409831 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:00.415031 master-1 kubenswrapper[4771]: W1011 10:56:00.414945 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0 WatchSource:0}: Error finding container c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0: Status 404 returned error can't find the container with id c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0 Oct 11 10:56:00.450576 master-1 kubenswrapper[4771]: I1011 10:56:00.450505 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6abca2-1aea-4da7-88aa-1d7651959165" path="/var/lib/kubelet/pods/0f6abca2-1aea-4da7-88aa-1d7651959165/volumes" Oct 11 10:56:00.537709 master-1 kubenswrapper[4771]: I1011 10:56:00.537592 4771 generic.go:334] "Generic (PLEG): container finished" podID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerID="9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e" exitCode=143 Oct 11 10:56:00.537709 master-1 kubenswrapper[4771]: I1011 10:56:00.537656 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"2deabbe8-397d-495c-aef9-afe91b4e9eeb","Type":"ContainerDied","Data":"9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e"} Oct 11 10:56:00.539637 master-1 kubenswrapper[4771]: I1011 10:56:00.539609 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-4bbqs" event={"ID":"5921e565-c581-42f4-8da8-df72fae9a3c0","Type":"ContainerStarted","Data":"f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded"} Oct 11 10:56:00.558515 master-1 kubenswrapper[4771]: I1011 10:56:00.551555 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerStarted","Data":"c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0"} Oct 11 10:56:01.438393 master-1 kubenswrapper[4771]: I1011 10:56:01.438298 4771 scope.go:117] "RemoveContainer" containerID="d389be566649d008a3e09b2815ff87839b176862458587a6a08bb0703f09d204" Oct 11 10:56:01.565654 master-1 kubenswrapper[4771]: I1011 10:56:01.565584 4771 generic.go:334] "Generic (PLEG): container finished" podID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerID="f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded" exitCode=0 Oct 11 10:56:01.565940 master-1 kubenswrapper[4771]: I1011 10:56:01.565710 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-4bbqs" event={"ID":"5921e565-c581-42f4-8da8-df72fae9a3c0","Type":"ContainerDied","Data":"f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded"} Oct 11 10:56:01.568582 master-1 kubenswrapper[4771]: I1011 10:56:01.568511 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerStarted","Data":"2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536"} Oct 11 10:56:02.529298 master-1 kubenswrapper[4771]: I1011 10:56:02.529210 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/glance-b5802-default-internal-api-1" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-httpd" probeResult="failure" output="Get \"https://10.129.0.130:9292/healthcheck\": read tcp 10.129.0.2:35066->10.129.0.130:9292: read: connection reset by peer" Oct 11 10:56:02.529298 master-1 kubenswrapper[4771]: I1011 10:56:02.529264 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/glance-b5802-default-internal-api-1" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-log" probeResult="failure" output="Get \"https://10.129.0.130:9292/healthcheck\": read tcp 10.129.0.2:35072->10.129.0.130:9292: read: connection reset by peer" Oct 11 10:56:02.588679 master-1 kubenswrapper[4771]: I1011 10:56:02.588621 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-4bbqs" event={"ID":"5921e565-c581-42f4-8da8-df72fae9a3c0","Type":"ContainerStarted","Data":"fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53"} Oct 11 10:56:02.597329 master-1 kubenswrapper[4771]: I1011 10:56:02.597073 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" event={"ID":"c6af8eba-f8bf-47f6-8313-7a902aeb170f","Type":"ContainerStarted","Data":"706cb456d30744ac062a8bfb63467464baef222596394937d45b40e61aa97f06"} Oct 11 10:56:02.598528 master-1 kubenswrapper[4771]: I1011 10:56:02.598476 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:56:02.611955 master-1 kubenswrapper[4771]: I1011 10:56:02.611812 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerStarted","Data":"6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111"} Oct 11 10:56:02.634864 master-1 kubenswrapper[4771]: I1011 10:56:02.634707 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/community-operators-4bbqs" podStartSLOduration=7.177732426 podStartE2EDuration="9.634671788s" podCreationTimestamp="2025-10-11 10:55:53 +0000 UTC" firstStartedPulling="2025-10-11 10:55:59.517524334 +0000 UTC m=+1791.491750775" lastFinishedPulling="2025-10-11 10:56:01.974463676 +0000 UTC m=+1793.948690137" observedRunningTime="2025-10-11 10:56:02.617268751 +0000 UTC m=+1794.591495192" watchObservedRunningTime="2025-10-11 10:56:02.634671788 +0000 UTC m=+1794.608898229" Oct 11 10:56:03.056981 master-1 kubenswrapper[4771]: I1011 10:56:03.056917 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.176596 master-1 kubenswrapper[4771]: I1011 10:56:03.176390 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-config-data\") pod \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " Oct 11 10:56:03.177970 master-1 kubenswrapper[4771]: I1011 10:56:03.177132 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g2z9\" (UniqueName: \"kubernetes.io/projected/2deabbe8-397d-495c-aef9-afe91b4e9eeb-kube-api-access-2g2z9\") pod \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " Oct 11 10:56:03.177970 master-1 kubenswrapper[4771]: I1011 10:56:03.177239 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-httpd-run\") pod \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " Oct 11 10:56:03.177970 master-1 kubenswrapper[4771]: I1011 10:56:03.177910 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-combined-ca-bundle\") pod \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " Oct 11 10:56:03.178124 master-1 kubenswrapper[4771]: I1011 10:56:03.178052 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"glance\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " Oct 11 10:56:03.178294 master-1 kubenswrapper[4771]: I1011 10:56:03.178210 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-logs\") pod \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " Oct 11 10:56:03.179203 master-1 kubenswrapper[4771]: I1011 10:56:03.178345 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-scripts\") pod \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " Oct 11 10:56:03.179203 master-1 kubenswrapper[4771]: I1011 10:56:03.178414 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-internal-tls-certs\") pod \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\" (UID: \"2deabbe8-397d-495c-aef9-afe91b4e9eeb\") " Oct 11 10:56:03.179203 master-1 kubenswrapper[4771]: I1011 10:56:03.179156 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-logs" (OuterVolumeSpecName: "logs") pod "2deabbe8-397d-495c-aef9-afe91b4e9eeb" (UID: "2deabbe8-397d-495c-aef9-afe91b4e9eeb"). InnerVolumeSpecName "logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:03.179488 master-1 kubenswrapper[4771]: I1011 10:56:03.179423 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-httpd-run" (OuterVolumeSpecName: "httpd-run") pod "2deabbe8-397d-495c-aef9-afe91b4e9eeb" (UID: "2deabbe8-397d-495c-aef9-afe91b4e9eeb"). InnerVolumeSpecName "httpd-run". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:03.180109 master-1 kubenswrapper[4771]: I1011 10:56:03.179451 4771 reconciler_common.go:293] "Volume detached for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-logs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:03.180519 master-1 kubenswrapper[4771]: I1011 10:56:03.180376 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2deabbe8-397d-495c-aef9-afe91b4e9eeb-kube-api-access-2g2z9" (OuterVolumeSpecName: "kube-api-access-2g2z9") pod "2deabbe8-397d-495c-aef9-afe91b4e9eeb" (UID: "2deabbe8-397d-495c-aef9-afe91b4e9eeb"). InnerVolumeSpecName "kube-api-access-2g2z9". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:03.182332 master-1 kubenswrapper[4771]: I1011 10:56:03.182247 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-scripts" (OuterVolumeSpecName: "scripts") pod "2deabbe8-397d-495c-aef9-afe91b4e9eeb" (UID: "2deabbe8-397d-495c-aef9-afe91b4e9eeb"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:03.201092 master-1 kubenswrapper[4771]: I1011 10:56:03.201001 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37" (OuterVolumeSpecName: "glance") pod "2deabbe8-397d-495c-aef9-afe91b4e9eeb" (UID: "2deabbe8-397d-495c-aef9-afe91b4e9eeb"). InnerVolumeSpecName "pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76". PluginName "kubernetes.io/csi", VolumeGidValue "" Oct 11 10:56:03.207470 master-1 kubenswrapper[4771]: I1011 10:56:03.207396 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "2deabbe8-397d-495c-aef9-afe91b4e9eeb" (UID: "2deabbe8-397d-495c-aef9-afe91b4e9eeb"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:03.234030 master-1 kubenswrapper[4771]: I1011 10:56:03.233951 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-config-data" (OuterVolumeSpecName: "config-data") pod "2deabbe8-397d-495c-aef9-afe91b4e9eeb" (UID: "2deabbe8-397d-495c-aef9-afe91b4e9eeb"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:03.241496 master-1 kubenswrapper[4771]: I1011 10:56:03.241423 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-internal-tls-certs" (OuterVolumeSpecName: "internal-tls-certs") pod "2deabbe8-397d-495c-aef9-afe91b4e9eeb" (UID: "2deabbe8-397d-495c-aef9-afe91b4e9eeb"). InnerVolumeSpecName "internal-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:03.285682 master-1 kubenswrapper[4771]: I1011 10:56:03.285588 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-2g2z9\" (UniqueName: \"kubernetes.io/projected/2deabbe8-397d-495c-aef9-afe91b4e9eeb-kube-api-access-2g2z9\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:03.285682 master-1 kubenswrapper[4771]: I1011 10:56:03.285685 4771 reconciler_common.go:293] "Volume detached for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/2deabbe8-397d-495c-aef9-afe91b4e9eeb-httpd-run\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:03.285682 master-1 kubenswrapper[4771]: I1011 10:56:03.285700 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:03.286115 master-1 kubenswrapper[4771]: I1011 10:56:03.285768 4771 reconciler_common.go:286] "operationExecutor.UnmountDevice started for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") on node \"master-1\" " Oct 11 10:56:03.286115 master-1 kubenswrapper[4771]: I1011 10:56:03.285782 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:03.286115 master-1 kubenswrapper[4771]: I1011 10:56:03.285792 4771 reconciler_common.go:293] "Volume detached for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-internal-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:03.286115 master-1 kubenswrapper[4771]: I1011 10:56:03.285805 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2deabbe8-397d-495c-aef9-afe91b4e9eeb-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:03.306668 master-1 kubenswrapper[4771]: I1011 10:56:03.306597 4771 csi_attacher.go:630] kubernetes.io/csi: attacher.UnmountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping UnmountDevice... Oct 11 10:56:03.306966 master-1 kubenswrapper[4771]: I1011 10:56:03.306935 4771 operation_generator.go:917] UnmountDevice succeeded for volume "pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76" (UniqueName: "kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37") on node "master-1" Oct 11 10:56:03.393440 master-1 kubenswrapper[4771]: I1011 10:56:03.388803 4771 reconciler_common.go:293] "Volume detached for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:03.625691 master-1 kubenswrapper[4771]: I1011 10:56:03.625602 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerStarted","Data":"dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68"} Oct 11 10:56:03.628441 master-1 kubenswrapper[4771]: I1011 10:56:03.628400 4771 generic.go:334] "Generic (PLEG): container finished" podID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerID="20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57" exitCode=0 Oct 11 10:56:03.629668 master-1 kubenswrapper[4771]: I1011 10:56:03.629646 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"2deabbe8-397d-495c-aef9-afe91b4e9eeb","Type":"ContainerDied","Data":"20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57"} Oct 11 10:56:03.629774 master-1 kubenswrapper[4771]: I1011 10:56:03.629761 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"2deabbe8-397d-495c-aef9-afe91b4e9eeb","Type":"ContainerDied","Data":"f7c97ec5b3e0ca2c1b2ecfb01745e5105213a6f53e9a75590979cbcd8d5e7e3f"} Oct 11 10:56:03.629868 master-1 kubenswrapper[4771]: I1011 10:56:03.629818 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.630151 master-1 kubenswrapper[4771]: I1011 10:56:03.629844 4771 scope.go:117] "RemoveContainer" containerID="20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57" Oct 11 10:56:03.660271 master-1 kubenswrapper[4771]: I1011 10:56:03.660226 4771 scope.go:117] "RemoveContainer" containerID="9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e" Oct 11 10:56:03.680141 master-1 kubenswrapper[4771]: I1011 10:56:03.680075 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:56:03.689414 master-1 kubenswrapper[4771]: I1011 10:56:03.689365 4771 scope.go:117] "RemoveContainer" containerID="20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57" Oct 11 10:56:03.689698 master-1 kubenswrapper[4771]: I1011 10:56:03.689676 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:56:03.690110 master-1 kubenswrapper[4771]: E1011 10:56:03.690087 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57\": container with ID starting with 20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57 not found: ID does not exist" containerID="20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57" Oct 11 10:56:03.690223 master-1 kubenswrapper[4771]: I1011 10:56:03.690195 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57"} err="failed to get container status \"20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57\": rpc error: code = NotFound desc = could not find container \"20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57\": container with ID starting with 20c69b75b312956ea0de71da2fddb27d441722583daf44724022755b0dd0fa57 not found: ID does not exist" Oct 11 10:56:03.690320 master-1 kubenswrapper[4771]: I1011 10:56:03.690305 4771 scope.go:117] "RemoveContainer" containerID="9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e" Oct 11 10:56:03.691005 master-1 kubenswrapper[4771]: E1011 10:56:03.690963 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e\": container with ID starting with 9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e not found: ID does not exist" containerID="9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e" Oct 11 10:56:03.691064 master-1 kubenswrapper[4771]: I1011 10:56:03.691023 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e"} err="failed to get container status \"9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e\": rpc error: code = NotFound desc = could not find container \"9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e\": container with ID starting with 9b109114c918b22d9cdecdecec8567888df427a39c7ae80e11cc66d73a78ae7e not found: ID does not exist" Oct 11 10:56:03.722200 master-1 kubenswrapper[4771]: I1011 10:56:03.722081 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:56:03.722928 master-1 kubenswrapper[4771]: E1011 10:56:03.722530 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-log" Oct 11 10:56:03.722928 master-1 kubenswrapper[4771]: I1011 10:56:03.722545 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-log" Oct 11 10:56:03.722928 master-1 kubenswrapper[4771]: E1011 10:56:03.722560 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-httpd" Oct 11 10:56:03.722928 master-1 kubenswrapper[4771]: I1011 10:56:03.722567 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-httpd" Oct 11 10:56:03.722928 master-1 kubenswrapper[4771]: I1011 10:56:03.722736 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-log" Oct 11 10:56:03.722928 master-1 kubenswrapper[4771]: I1011 10:56:03.722763 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" containerName="glance-httpd" Oct 11 10:56:03.723998 master-1 kubenswrapper[4771]: I1011 10:56:03.723975 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.727379 master-1 kubenswrapper[4771]: I1011 10:56:03.727310 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"glance-b5802-default-internal-config-data" Oct 11 10:56:03.727648 master-1 kubenswrapper[4771]: I1011 10:56:03.727629 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-glance-default-internal-svc" Oct 11 10:56:03.742573 master-1 kubenswrapper[4771]: I1011 10:56:03.742509 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:56:03.797779 master-1 kubenswrapper[4771]: I1011 10:56:03.797687 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-config-data\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.798144 master-1 kubenswrapper[4771]: I1011 10:56:03.798032 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-internal-tls-certs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.798321 master-1 kubenswrapper[4771]: I1011 10:56:03.798288 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.798657 master-1 kubenswrapper[4771]: I1011 10:56:03.798627 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/2cc63350-c31a-459f-b45b-73f465e53bc5-logs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.798980 master-1 kubenswrapper[4771]: I1011 10:56:03.798954 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-scripts\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.799040 master-1 kubenswrapper[4771]: I1011 10:56:03.799005 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/2cc63350-c31a-459f-b45b-73f465e53bc5-httpd-run\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.799040 master-1 kubenswrapper[4771]: I1011 10:56:03.799032 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-combined-ca-bundle\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.799110 master-1 kubenswrapper[4771]: I1011 10:56:03.799087 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9xdn\" (UniqueName: \"kubernetes.io/projected/2cc63350-c31a-459f-b45b-73f465e53bc5-kube-api-access-x9xdn\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.900950 master-1 kubenswrapper[4771]: I1011 10:56:03.900774 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.901398 master-1 kubenswrapper[4771]: I1011 10:56:03.901348 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/2cc63350-c31a-459f-b45b-73f465e53bc5-logs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.901565 master-1 kubenswrapper[4771]: I1011 10:56:03.901548 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-scripts\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.901685 master-1 kubenswrapper[4771]: I1011 10:56:03.901665 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/2cc63350-c31a-459f-b45b-73f465e53bc5-httpd-run\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.901813 master-1 kubenswrapper[4771]: I1011 10:56:03.901789 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-combined-ca-bundle\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.901956 master-1 kubenswrapper[4771]: I1011 10:56:03.901936 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-x9xdn\" (UniqueName: \"kubernetes.io/projected/2cc63350-c31a-459f-b45b-73f465e53bc5-kube-api-access-x9xdn\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.902110 master-1 kubenswrapper[4771]: I1011 10:56:03.902089 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-config-data\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.902332 master-1 kubenswrapper[4771]: I1011 10:56:03.902273 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/2cc63350-c31a-459f-b45b-73f465e53bc5-logs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.902441 master-1 kubenswrapper[4771]: I1011 10:56:03.902405 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/2cc63350-c31a-459f-b45b-73f465e53bc5-httpd-run\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.902512 master-1 kubenswrapper[4771]: I1011 10:56:03.902495 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-internal-tls-certs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.904899 master-1 kubenswrapper[4771]: I1011 10:56:03.904848 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:56:03.905005 master-1 kubenswrapper[4771]: I1011 10:56:03.904916 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/319ddbbf14dc29e9dbd7eec9a997b70a9a11c6eca7f6496495d34ea4ac3ccad0/globalmount\"" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.905939 master-1 kubenswrapper[4771]: I1011 10:56:03.905885 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-scripts\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.907769 master-1 kubenswrapper[4771]: I1011 10:56:03.907735 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-config-data\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.908467 master-1 kubenswrapper[4771]: I1011 10:56:03.908417 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-combined-ca-bundle\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.909251 master-1 kubenswrapper[4771]: I1011 10:56:03.909224 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cc63350-c31a-459f-b45b-73f465e53bc5-internal-tls-certs\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:03.933642 master-1 kubenswrapper[4771]: I1011 10:56:03.933560 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9xdn\" (UniqueName: \"kubernetes.io/projected/2cc63350-c31a-459f-b45b-73f465e53bc5-kube-api-access-x9xdn\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:04.275665 master-1 kubenswrapper[4771]: I1011 10:56:04.275600 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:56:04.275830 master-1 kubenswrapper[4771]: I1011 10:56:04.275801 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:56:04.451869 master-1 kubenswrapper[4771]: I1011 10:56:04.451709 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2deabbe8-397d-495c-aef9-afe91b4e9eeb" path="/var/lib/kubelet/pods/2deabbe8-397d-495c-aef9-afe91b4e9eeb/volumes" Oct 11 10:56:04.642679 master-1 kubenswrapper[4771]: I1011 10:56:04.642623 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerStarted","Data":"505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe"} Oct 11 10:56:04.643199 master-1 kubenswrapper[4771]: I1011 10:56:04.642930 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 10:56:04.687378 master-1 kubenswrapper[4771]: I1011 10:56:04.686309 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=2.076279791 podStartE2EDuration="5.686280709s" podCreationTimestamp="2025-10-11 10:55:59 +0000 UTC" firstStartedPulling="2025-10-11 10:56:00.417989474 +0000 UTC m=+1792.392215915" lastFinishedPulling="2025-10-11 10:56:04.027990392 +0000 UTC m=+1796.002216833" observedRunningTime="2025-10-11 10:56:04.675044252 +0000 UTC m=+1796.649270753" watchObservedRunningTime="2025-10-11 10:56:04.686280709 +0000 UTC m=+1796.660507190" Oct 11 10:56:05.088286 master-1 kubenswrapper[4771]: I1011 10:56:05.088187 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-b6ba1d46-4dc8-46c7-b2c2-511158bb8b76\" (UniqueName: \"kubernetes.io/csi/topolvm.io^ed627bc4-642c-44b9-a346-0d8fa1903b37\") pod \"glance-b5802-default-internal-api-1\" (UID: \"2cc63350-c31a-459f-b45b-73f465e53bc5\") " pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:05.242331 master-1 kubenswrapper[4771]: I1011 10:56:05.242181 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:05.321383 master-1 kubenswrapper[4771]: I1011 10:56:05.318912 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ironic-neutron-agent-656ddc8b67-kfkzr" Oct 11 10:56:05.325380 master-1 kubenswrapper[4771]: I1011 10:56:05.322761 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-marketplace/community-operators-4bbqs" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerName="registry-server" probeResult="failure" output=< Oct 11 10:56:05.325380 master-1 kubenswrapper[4771]: timeout: failed to connect service ":50051" within 1s Oct 11 10:56:05.325380 master-1 kubenswrapper[4771]: > Oct 11 10:56:05.854822 master-1 kubenswrapper[4771]: W1011 10:56:05.851129 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cc63350_c31a_459f_b45b_73f465e53bc5.slice/crio-ea4cb66a3c0705f4d1840f224e02cd6bc0111d2c882543d0a0d1e1ac22aefe85 WatchSource:0}: Error finding container ea4cb66a3c0705f4d1840f224e02cd6bc0111d2c882543d0a0d1e1ac22aefe85: Status 404 returned error can't find the container with id ea4cb66a3c0705f4d1840f224e02cd6bc0111d2c882543d0a0d1e1ac22aefe85 Oct 11 10:56:05.862626 master-1 kubenswrapper[4771]: I1011 10:56:05.862411 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-internal-api-1"] Oct 11 10:56:06.665862 master-1 kubenswrapper[4771]: I1011 10:56:06.665649 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"2cc63350-c31a-459f-b45b-73f465e53bc5","Type":"ContainerStarted","Data":"f389296367a4d076b95e26ae48f492e36a8ecbfcb2d24a4e1b0c219f613e5043"} Oct 11 10:56:06.665862 master-1 kubenswrapper[4771]: I1011 10:56:06.665747 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"2cc63350-c31a-459f-b45b-73f465e53bc5","Type":"ContainerStarted","Data":"ea4cb66a3c0705f4d1840f224e02cd6bc0111d2c882543d0a0d1e1ac22aefe85"} Oct 11 10:56:07.651885 master-1 kubenswrapper[4771]: I1011 10:56:07.651734 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/aodh-db-create-sz8dm"] Oct 11 10:56:07.654443 master-1 kubenswrapper[4771]: I1011 10:56:07.654390 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-db-create-sz8dm" Oct 11 10:56:07.682842 master-1 kubenswrapper[4771]: I1011 10:56:07.682762 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/aodh-db-create-sz8dm"] Oct 11 10:56:07.700317 master-1 kubenswrapper[4771]: I1011 10:56:07.700206 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vch5\" (UniqueName: \"kubernetes.io/projected/fa024267-404c-497a-a798-3a371608b678-kube-api-access-5vch5\") pod \"aodh-db-create-sz8dm\" (UID: \"fa024267-404c-497a-a798-3a371608b678\") " pod="openstack/aodh-db-create-sz8dm" Oct 11 10:56:07.727110 master-1 kubenswrapper[4771]: I1011 10:56:07.727020 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-internal-api-1" event={"ID":"2cc63350-c31a-459f-b45b-73f465e53bc5","Type":"ContainerStarted","Data":"a0dc956d5f2ab55e2217e32912c7cbe0043299962352ddb238b696672680ac68"} Oct 11 10:56:07.802525 master-1 kubenswrapper[4771]: I1011 10:56:07.801964 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5vch5\" (UniqueName: \"kubernetes.io/projected/fa024267-404c-497a-a798-3a371608b678-kube-api-access-5vch5\") pod \"aodh-db-create-sz8dm\" (UID: \"fa024267-404c-497a-a798-3a371608b678\") " pod="openstack/aodh-db-create-sz8dm" Oct 11 10:56:07.968927 master-1 kubenswrapper[4771]: I1011 10:56:07.968752 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vch5\" (UniqueName: \"kubernetes.io/projected/fa024267-404c-497a-a798-3a371608b678-kube-api-access-5vch5\") pod \"aodh-db-create-sz8dm\" (UID: \"fa024267-404c-497a-a798-3a371608b678\") " pod="openstack/aodh-db-create-sz8dm" Oct 11 10:56:07.975682 master-1 kubenswrapper[4771]: I1011 10:56:07.975630 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-db-create-sz8dm" Oct 11 10:56:08.000216 master-1 kubenswrapper[4771]: I1011 10:56:07.999255 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/glance-b5802-default-internal-api-1" podStartSLOduration=4.999216864 podStartE2EDuration="4.999216864s" podCreationTimestamp="2025-10-11 10:56:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:56:07.997513464 +0000 UTC m=+1799.971739935" watchObservedRunningTime="2025-10-11 10:56:07.999216864 +0000 UTC m=+1799.973443345" Oct 11 10:56:08.464645 master-1 kubenswrapper[4771]: I1011 10:56:08.464567 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/aodh-db-create-sz8dm"] Oct 11 10:56:08.471820 master-1 kubenswrapper[4771]: W1011 10:56:08.471742 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa024267_404c_497a_a798_3a371608b678.slice/crio-abbe609448c0eeb6fc32b8a0fd7ff7adaccda13bac04e7607b9d744d2c05e950 WatchSource:0}: Error finding container abbe609448c0eeb6fc32b8a0fd7ff7adaccda13bac04e7607b9d744d2c05e950: Status 404 returned error can't find the container with id abbe609448c0eeb6fc32b8a0fd7ff7adaccda13bac04e7607b9d744d2c05e950 Oct 11 10:56:08.743109 master-1 kubenswrapper[4771]: I1011 10:56:08.740517 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-db-create-sz8dm" event={"ID":"fa024267-404c-497a-a798-3a371608b678","Type":"ContainerStarted","Data":"558ef049f7336b936f032c4d3e3115131e36703eb572e93323b57a5fd484ff9e"} Oct 11 10:56:08.743109 master-1 kubenswrapper[4771]: I1011 10:56:08.740611 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-db-create-sz8dm" event={"ID":"fa024267-404c-497a-a798-3a371608b678","Type":"ContainerStarted","Data":"abbe609448c0eeb6fc32b8a0fd7ff7adaccda13bac04e7607b9d744d2c05e950"} Oct 11 10:56:08.781008 master-1 kubenswrapper[4771]: I1011 10:56:08.780767 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/aodh-db-create-sz8dm" podStartSLOduration=1.780739235 podStartE2EDuration="1.780739235s" podCreationTimestamp="2025-10-11 10:56:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:56:08.767294843 +0000 UTC m=+1800.741521294" watchObservedRunningTime="2025-10-11 10:56:08.780739235 +0000 UTC m=+1800.754965686" Oct 11 10:56:09.443106 master-1 kubenswrapper[4771]: I1011 10:56:09.441347 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/cinder-b5802-api-0"] Oct 11 10:56:09.443106 master-1 kubenswrapper[4771]: I1011 10:56:09.441887 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/cinder-b5802-api-0" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerName="cinder-b5802-api-log" containerID="cri-o://fff01313b302342cb30f2201b57bb76a5615b3d6076b484b8fb9b7d061e529af" gracePeriod=30 Oct 11 10:56:09.443106 master-1 kubenswrapper[4771]: I1011 10:56:09.442163 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/cinder-b5802-api-0" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerName="cinder-api" containerID="cri-o://cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80" gracePeriod=30 Oct 11 10:56:09.749538 master-1 kubenswrapper[4771]: I1011 10:56:09.749350 4771 generic.go:334] "Generic (PLEG): container finished" podID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerID="fff01313b302342cb30f2201b57bb76a5615b3d6076b484b8fb9b7d061e529af" exitCode=143 Oct 11 10:56:09.749538 master-1 kubenswrapper[4771]: I1011 10:56:09.749420 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/cinder-b5802-api-0" event={"ID":"478147ef-a0d7-4c37-952c-3fc3a23775db","Type":"ContainerDied","Data":"fff01313b302342cb30f2201b57bb76a5615b3d6076b484b8fb9b7d061e529af"} Oct 11 10:56:09.751517 master-1 kubenswrapper[4771]: I1011 10:56:09.751449 4771 generic.go:334] "Generic (PLEG): container finished" podID="fa024267-404c-497a-a798-3a371608b678" containerID="558ef049f7336b936f032c4d3e3115131e36703eb572e93323b57a5fd484ff9e" exitCode=0 Oct 11 10:56:09.751618 master-1 kubenswrapper[4771]: I1011 10:56:09.751516 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-db-create-sz8dm" event={"ID":"fa024267-404c-497a-a798-3a371608b678","Type":"ContainerDied","Data":"558ef049f7336b936f032c4d3e3115131e36703eb572e93323b57a5fd484ff9e"} Oct 11 10:56:11.320495 master-1 kubenswrapper[4771]: I1011 10:56:11.320427 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-db-create-sz8dm" Oct 11 10:56:11.396073 master-1 kubenswrapper[4771]: I1011 10:56:11.395968 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vch5\" (UniqueName: \"kubernetes.io/projected/fa024267-404c-497a-a798-3a371608b678-kube-api-access-5vch5\") pod \"fa024267-404c-497a-a798-3a371608b678\" (UID: \"fa024267-404c-497a-a798-3a371608b678\") " Oct 11 10:56:11.399723 master-1 kubenswrapper[4771]: I1011 10:56:11.399646 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa024267-404c-497a-a798-3a371608b678-kube-api-access-5vch5" (OuterVolumeSpecName: "kube-api-access-5vch5") pod "fa024267-404c-497a-a798-3a371608b678" (UID: "fa024267-404c-497a-a798-3a371608b678"). InnerVolumeSpecName "kube-api-access-5vch5". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:11.498826 master-1 kubenswrapper[4771]: I1011 10:56:11.498770 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-5vch5\" (UniqueName: \"kubernetes.io/projected/fa024267-404c-497a-a798-3a371608b678-kube-api-access-5vch5\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:11.782980 master-1 kubenswrapper[4771]: I1011 10:56:11.782912 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-db-create-sz8dm" event={"ID":"fa024267-404c-497a-a798-3a371608b678","Type":"ContainerDied","Data":"abbe609448c0eeb6fc32b8a0fd7ff7adaccda13bac04e7607b9d744d2c05e950"} Oct 11 10:56:11.783346 master-1 kubenswrapper[4771]: I1011 10:56:11.783326 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abbe609448c0eeb6fc32b8a0fd7ff7adaccda13bac04e7607b9d744d2c05e950" Oct 11 10:56:11.783480 master-1 kubenswrapper[4771]: I1011 10:56:11.783006 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-db-create-sz8dm" Oct 11 10:56:12.588995 master-1 kubenswrapper[4771]: I1011 10:56:12.588897 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/cinder-b5802-api-0" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerName="cinder-api" probeResult="failure" output="Get \"http://10.129.0.132:8776/healthcheck\": read tcp 10.129.0.2:35862->10.129.0.132:8776: read: connection reset by peer" Oct 11 10:56:12.798206 master-1 kubenswrapper[4771]: I1011 10:56:12.798124 4771 generic.go:334] "Generic (PLEG): container finished" podID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerID="cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80" exitCode=0 Oct 11 10:56:12.799675 master-1 kubenswrapper[4771]: I1011 10:56:12.798228 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/cinder-b5802-api-0" event={"ID":"478147ef-a0d7-4c37-952c-3fc3a23775db","Type":"ContainerDied","Data":"cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80"} Oct 11 10:56:13.123535 master-1 kubenswrapper[4771]: I1011 10:56:13.123242 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.232565 master-1 kubenswrapper[4771]: I1011 10:56:13.232510 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/478147ef-a0d7-4c37-952c-3fc3a23775db-logs\") pod \"478147ef-a0d7-4c37-952c-3fc3a23775db\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " Oct 11 10:56:13.232565 master-1 kubenswrapper[4771]: I1011 10:56:13.232574 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-scripts\") pod \"478147ef-a0d7-4c37-952c-3fc3a23775db\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " Oct 11 10:56:13.233019 master-1 kubenswrapper[4771]: I1011 10:56:13.232753 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svvlr\" (UniqueName: \"kubernetes.io/projected/478147ef-a0d7-4c37-952c-3fc3a23775db-kube-api-access-svvlr\") pod \"478147ef-a0d7-4c37-952c-3fc3a23775db\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " Oct 11 10:56:13.233019 master-1 kubenswrapper[4771]: I1011 10:56:13.232793 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data\") pod \"478147ef-a0d7-4c37-952c-3fc3a23775db\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " Oct 11 10:56:13.233019 master-1 kubenswrapper[4771]: I1011 10:56:13.232860 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data-custom\") pod \"478147ef-a0d7-4c37-952c-3fc3a23775db\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " Oct 11 10:56:13.233019 master-1 kubenswrapper[4771]: I1011 10:56:13.232895 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"etc-machine-id\" (UniqueName: \"kubernetes.io/host-path/478147ef-a0d7-4c37-952c-3fc3a23775db-etc-machine-id\") pod \"478147ef-a0d7-4c37-952c-3fc3a23775db\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " Oct 11 10:56:13.233019 master-1 kubenswrapper[4771]: I1011 10:56:13.232972 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-combined-ca-bundle\") pod \"478147ef-a0d7-4c37-952c-3fc3a23775db\" (UID: \"478147ef-a0d7-4c37-952c-3fc3a23775db\") " Oct 11 10:56:13.233245 master-1 kubenswrapper[4771]: I1011 10:56:13.233148 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/478147ef-a0d7-4c37-952c-3fc3a23775db-etc-machine-id" (OuterVolumeSpecName: "etc-machine-id") pod "478147ef-a0d7-4c37-952c-3fc3a23775db" (UID: "478147ef-a0d7-4c37-952c-3fc3a23775db"). InnerVolumeSpecName "etc-machine-id". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 10:56:13.233245 master-1 kubenswrapper[4771]: I1011 10:56:13.233198 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478147ef-a0d7-4c37-952c-3fc3a23775db-logs" (OuterVolumeSpecName: "logs") pod "478147ef-a0d7-4c37-952c-3fc3a23775db" (UID: "478147ef-a0d7-4c37-952c-3fc3a23775db"). InnerVolumeSpecName "logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:13.233666 master-1 kubenswrapper[4771]: I1011 10:56:13.233630 4771 reconciler_common.go:293] "Volume detached for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/478147ef-a0d7-4c37-952c-3fc3a23775db-logs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:13.233782 master-1 kubenswrapper[4771]: I1011 10:56:13.233675 4771 reconciler_common.go:293] "Volume detached for volume \"etc-machine-id\" (UniqueName: \"kubernetes.io/host-path/478147ef-a0d7-4c37-952c-3fc3a23775db-etc-machine-id\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:13.236868 master-1 kubenswrapper[4771]: I1011 10:56:13.236819 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data-custom" (OuterVolumeSpecName: "config-data-custom") pod "478147ef-a0d7-4c37-952c-3fc3a23775db" (UID: "478147ef-a0d7-4c37-952c-3fc3a23775db"). InnerVolumeSpecName "config-data-custom". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:13.237657 master-1 kubenswrapper[4771]: I1011 10:56:13.237614 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478147ef-a0d7-4c37-952c-3fc3a23775db-kube-api-access-svvlr" (OuterVolumeSpecName: "kube-api-access-svvlr") pod "478147ef-a0d7-4c37-952c-3fc3a23775db" (UID: "478147ef-a0d7-4c37-952c-3fc3a23775db"). InnerVolumeSpecName "kube-api-access-svvlr". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:13.248786 master-1 kubenswrapper[4771]: I1011 10:56:13.248690 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-scripts" (OuterVolumeSpecName: "scripts") pod "478147ef-a0d7-4c37-952c-3fc3a23775db" (UID: "478147ef-a0d7-4c37-952c-3fc3a23775db"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:13.257346 master-1 kubenswrapper[4771]: I1011 10:56:13.257251 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "478147ef-a0d7-4c37-952c-3fc3a23775db" (UID: "478147ef-a0d7-4c37-952c-3fc3a23775db"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:13.277430 master-1 kubenswrapper[4771]: I1011 10:56:13.277331 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data" (OuterVolumeSpecName: "config-data") pod "478147ef-a0d7-4c37-952c-3fc3a23775db" (UID: "478147ef-a0d7-4c37-952c-3fc3a23775db"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:13.335883 master-1 kubenswrapper[4771]: I1011 10:56:13.335783 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-svvlr\" (UniqueName: \"kubernetes.io/projected/478147ef-a0d7-4c37-952c-3fc3a23775db-kube-api-access-svvlr\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:13.335883 master-1 kubenswrapper[4771]: I1011 10:56:13.335848 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:13.335883 master-1 kubenswrapper[4771]: I1011 10:56:13.335859 4771 reconciler_common.go:293] "Volume detached for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-config-data-custom\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:13.335883 master-1 kubenswrapper[4771]: I1011 10:56:13.335872 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:13.335883 master-1 kubenswrapper[4771]: I1011 10:56:13.335882 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/478147ef-a0d7-4c37-952c-3fc3a23775db-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:13.809497 master-1 kubenswrapper[4771]: I1011 10:56:13.809382 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/cinder-b5802-api-0" event={"ID":"478147ef-a0d7-4c37-952c-3fc3a23775db","Type":"ContainerDied","Data":"48979e1f4a2c3ea1ab7c6b56d19ad482a69e0c44021b08cc343a04080547c276"} Oct 11 10:56:13.809497 master-1 kubenswrapper[4771]: I1011 10:56:13.809493 4771 scope.go:117] "RemoveContainer" containerID="cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80" Oct 11 10:56:13.810701 master-1 kubenswrapper[4771]: I1011 10:56:13.810642 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.838945 master-1 kubenswrapper[4771]: I1011 10:56:13.838815 4771 scope.go:117] "RemoveContainer" containerID="fff01313b302342cb30f2201b57bb76a5615b3d6076b484b8fb9b7d061e529af" Oct 11 10:56:13.860120 master-1 kubenswrapper[4771]: I1011 10:56:13.860062 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/cinder-b5802-api-0"] Oct 11 10:56:13.866233 master-1 kubenswrapper[4771]: I1011 10:56:13.866171 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/cinder-b5802-api-0"] Oct 11 10:56:13.898958 master-1 kubenswrapper[4771]: I1011 10:56:13.898887 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/cinder-b5802-api-0"] Oct 11 10:56:13.899303 master-1 kubenswrapper[4771]: E1011 10:56:13.899274 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="fa024267-404c-497a-a798-3a371608b678" containerName="mariadb-database-create" Oct 11 10:56:13.899303 master-1 kubenswrapper[4771]: I1011 10:56:13.899294 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa024267-404c-497a-a798-3a371608b678" containerName="mariadb-database-create" Oct 11 10:56:13.899398 master-1 kubenswrapper[4771]: E1011 10:56:13.899317 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerName="cinder-api" Oct 11 10:56:13.899398 master-1 kubenswrapper[4771]: I1011 10:56:13.899325 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerName="cinder-api" Oct 11 10:56:13.899398 master-1 kubenswrapper[4771]: E1011 10:56:13.899342 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerName="cinder-b5802-api-log" Oct 11 10:56:13.899398 master-1 kubenswrapper[4771]: I1011 10:56:13.899365 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerName="cinder-b5802-api-log" Oct 11 10:56:13.899526 master-1 kubenswrapper[4771]: I1011 10:56:13.899491 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerName="cinder-api" Oct 11 10:56:13.899526 master-1 kubenswrapper[4771]: I1011 10:56:13.899512 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" containerName="cinder-b5802-api-log" Oct 11 10:56:13.899526 master-1 kubenswrapper[4771]: I1011 10:56:13.899524 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="fa024267-404c-497a-a798-3a371608b678" containerName="mariadb-database-create" Oct 11 10:56:13.900770 master-1 kubenswrapper[4771]: I1011 10:56:13.900739 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.904691 master-1 kubenswrapper[4771]: I1011 10:56:13.904659 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cinder-b5802-api-config-data" Oct 11 10:56:13.904774 master-1 kubenswrapper[4771]: I1011 10:56:13.904691 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cinder-b5802-scripts" Oct 11 10:56:13.904774 master-1 kubenswrapper[4771]: I1011 10:56:13.904696 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-cinder-internal-svc" Oct 11 10:56:13.904924 master-1 kubenswrapper[4771]: I1011 10:56:13.904895 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-cinder-public-svc" Oct 11 10:56:13.905026 master-1 kubenswrapper[4771]: I1011 10:56:13.905003 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cinder-b5802-config-data" Oct 11 10:56:13.925589 master-1 kubenswrapper[4771]: I1011 10:56:13.925534 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/cinder-b5802-api-0"] Oct 11 10:56:13.950911 master-1 kubenswrapper[4771]: I1011 10:56:13.950824 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-combined-ca-bundle\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.951110 master-1 kubenswrapper[4771]: I1011 10:56:13.950948 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-config-data-custom\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.951110 master-1 kubenswrapper[4771]: I1011 10:56:13.951005 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-internal-tls-certs\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.951110 master-1 kubenswrapper[4771]: I1011 10:56:13.951049 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-public-tls-certs\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.951211 master-1 kubenswrapper[4771]: I1011 10:56:13.951119 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sc9t\" (UniqueName: \"kubernetes.io/projected/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-kube-api-access-4sc9t\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.951249 master-1 kubenswrapper[4771]: I1011 10:56:13.951214 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-config-data\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.951289 master-1 kubenswrapper[4771]: I1011 10:56:13.951268 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-logs\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.951333 master-1 kubenswrapper[4771]: I1011 10:56:13.951308 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-machine-id\" (UniqueName: \"kubernetes.io/host-path/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-etc-machine-id\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:13.951385 master-1 kubenswrapper[4771]: I1011 10:56:13.951347 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-scripts\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.053221 master-1 kubenswrapper[4771]: I1011 10:56:14.053147 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-scripts\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.053534 master-1 kubenswrapper[4771]: I1011 10:56:14.053239 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-combined-ca-bundle\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.053534 master-1 kubenswrapper[4771]: I1011 10:56:14.053307 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-config-data-custom\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.053534 master-1 kubenswrapper[4771]: I1011 10:56:14.053397 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-internal-tls-certs\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.053534 master-1 kubenswrapper[4771]: I1011 10:56:14.053457 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-public-tls-certs\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.053534 master-1 kubenswrapper[4771]: I1011 10:56:14.053495 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-4sc9t\" (UniqueName: \"kubernetes.io/projected/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-kube-api-access-4sc9t\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.053739 master-1 kubenswrapper[4771]: I1011 10:56:14.053592 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-config-data\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.053739 master-1 kubenswrapper[4771]: I1011 10:56:14.053654 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-logs\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.053739 master-1 kubenswrapper[4771]: I1011 10:56:14.053698 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"etc-machine-id\" (UniqueName: \"kubernetes.io/host-path/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-etc-machine-id\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.054119 master-1 kubenswrapper[4771]: I1011 10:56:14.053845 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"etc-machine-id\" (UniqueName: \"kubernetes.io/host-path/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-etc-machine-id\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.054728 master-1 kubenswrapper[4771]: I1011 10:56:14.054492 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-logs\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.058156 master-1 kubenswrapper[4771]: I1011 10:56:14.057627 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-config-data\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.058156 master-1 kubenswrapper[4771]: I1011 10:56:14.058100 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-public-tls-certs\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.058531 master-1 kubenswrapper[4771]: I1011 10:56:14.058482 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-internal-tls-certs\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.058604 master-1 kubenswrapper[4771]: I1011 10:56:14.058487 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-scripts\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.058894 master-1 kubenswrapper[4771]: I1011 10:56:14.058839 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-config-data-custom\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.060922 master-1 kubenswrapper[4771]: I1011 10:56:14.060852 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-combined-ca-bundle\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.094955 master-1 kubenswrapper[4771]: I1011 10:56:14.094868 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sc9t\" (UniqueName: \"kubernetes.io/projected/4a6945b7-6d08-4a4a-9627-5993eb5e0a7f-kube-api-access-4sc9t\") pod \"cinder-b5802-api-0\" (UID: \"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f\") " pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.225686 master-1 kubenswrapper[4771]: I1011 10:56:14.225617 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/cinder-b5802-api-0" Oct 11 10:56:14.327231 master-1 kubenswrapper[4771]: I1011 10:56:14.327157 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:56:14.414931 master-1 kubenswrapper[4771]: I1011 10:56:14.414499 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:56:14.459718 master-1 kubenswrapper[4771]: I1011 10:56:14.459632 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="478147ef-a0d7-4c37-952c-3fc3a23775db" path="/var/lib/kubelet/pods/478147ef-a0d7-4c37-952c-3fc3a23775db/volumes" Oct 11 10:56:14.953456 master-1 kubenswrapper[4771]: I1011 10:56:14.953299 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/cinder-b5802-api-0"] Oct 11 10:56:14.990681 master-1 kubenswrapper[4771]: I1011 10:56:14.990612 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-4bbqs"] Oct 11 10:56:15.243140 master-1 kubenswrapper[4771]: I1011 10:56:15.242973 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:15.243140 master-1 kubenswrapper[4771]: I1011 10:56:15.243073 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:15.287801 master-1 kubenswrapper[4771]: I1011 10:56:15.287524 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:15.302469 master-1 kubenswrapper[4771]: I1011 10:56:15.301900 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:15.829464 master-1 kubenswrapper[4771]: I1011 10:56:15.829398 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/cinder-b5802-api-0" event={"ID":"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f","Type":"ContainerStarted","Data":"f42ad3585d89b67cdab529dc67ac31a8c2d471a973b6fcbb6740c8dd37b76a96"} Oct 11 10:56:15.829464 master-1 kubenswrapper[4771]: I1011 10:56:15.829459 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/cinder-b5802-api-0" event={"ID":"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f","Type":"ContainerStarted","Data":"c69d5d53470534d9c412bdab403cbe0881354ba1e16ffac19d620e059bd28afb"} Oct 11 10:56:15.829949 master-1 kubenswrapper[4771]: I1011 10:56:15.829837 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:15.829949 master-1 kubenswrapper[4771]: I1011 10:56:15.829877 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:15.829949 master-1 kubenswrapper[4771]: I1011 10:56:15.829834 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/community-operators-4bbqs" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerName="registry-server" containerID="cri-o://fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53" gracePeriod=2 Oct 11 10:56:16.418322 master-1 kubenswrapper[4771]: I1011 10:56:16.418251 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:56:16.516654 master-1 kubenswrapper[4771]: I1011 10:56:16.515374 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-utilities\") pod \"5921e565-c581-42f4-8da8-df72fae9a3c0\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " Oct 11 10:56:16.516654 master-1 kubenswrapper[4771]: I1011 10:56:16.515702 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-catalog-content\") pod \"5921e565-c581-42f4-8da8-df72fae9a3c0\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " Oct 11 10:56:16.516654 master-1 kubenswrapper[4771]: I1011 10:56:16.515808 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzpqz\" (UniqueName: \"kubernetes.io/projected/5921e565-c581-42f4-8da8-df72fae9a3c0-kube-api-access-jzpqz\") pod \"5921e565-c581-42f4-8da8-df72fae9a3c0\" (UID: \"5921e565-c581-42f4-8da8-df72fae9a3c0\") " Oct 11 10:56:16.518185 master-1 kubenswrapper[4771]: I1011 10:56:16.517204 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-utilities" (OuterVolumeSpecName: "utilities") pod "5921e565-c581-42f4-8da8-df72fae9a3c0" (UID: "5921e565-c581-42f4-8da8-df72fae9a3c0"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:16.526466 master-1 kubenswrapper[4771]: I1011 10:56:16.525751 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5921e565-c581-42f4-8da8-df72fae9a3c0-kube-api-access-jzpqz" (OuterVolumeSpecName: "kube-api-access-jzpqz") pod "5921e565-c581-42f4-8da8-df72fae9a3c0" (UID: "5921e565-c581-42f4-8da8-df72fae9a3c0"). InnerVolumeSpecName "kube-api-access-jzpqz". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:16.576231 master-1 kubenswrapper[4771]: I1011 10:56:16.576163 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "5921e565-c581-42f4-8da8-df72fae9a3c0" (UID: "5921e565-c581-42f4-8da8-df72fae9a3c0"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:16.617766 master-1 kubenswrapper[4771]: I1011 10:56:16.617713 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-jzpqz\" (UniqueName: \"kubernetes.io/projected/5921e565-c581-42f4-8da8-df72fae9a3c0-kube-api-access-jzpqz\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:16.618019 master-1 kubenswrapper[4771]: I1011 10:56:16.618007 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:16.618099 master-1 kubenswrapper[4771]: I1011 10:56:16.618089 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/5921e565-c581-42f4-8da8-df72fae9a3c0-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:16.841136 master-1 kubenswrapper[4771]: I1011 10:56:16.841067 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/cinder-b5802-api-0" event={"ID":"4a6945b7-6d08-4a4a-9627-5993eb5e0a7f","Type":"ContainerStarted","Data":"af433967ea39199deedf56b2f1f2283de9b047e4635f3608fd9fbf9f788ee162"} Oct 11 10:56:16.841506 master-1 kubenswrapper[4771]: I1011 10:56:16.841220 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/cinder-b5802-api-0" Oct 11 10:56:16.844979 master-1 kubenswrapper[4771]: I1011 10:56:16.844928 4771 generic.go:334] "Generic (PLEG): container finished" podID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerID="fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53" exitCode=0 Oct 11 10:56:16.845160 master-1 kubenswrapper[4771]: I1011 10:56:16.845008 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-4bbqs" event={"ID":"5921e565-c581-42f4-8da8-df72fae9a3c0","Type":"ContainerDied","Data":"fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53"} Oct 11 10:56:16.845160 master-1 kubenswrapper[4771]: I1011 10:56:16.845083 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-4bbqs" event={"ID":"5921e565-c581-42f4-8da8-df72fae9a3c0","Type":"ContainerDied","Data":"2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe"} Oct 11 10:56:16.845292 master-1 kubenswrapper[4771]: I1011 10:56:16.845163 4771 scope.go:117] "RemoveContainer" containerID="fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53" Oct 11 10:56:16.845571 master-1 kubenswrapper[4771]: I1011 10:56:16.845540 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-4bbqs" Oct 11 10:56:16.868252 master-1 kubenswrapper[4771]: I1011 10:56:16.868209 4771 scope.go:117] "RemoveContainer" containerID="f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded" Oct 11 10:56:16.906471 master-1 kubenswrapper[4771]: I1011 10:56:16.906172 4771 scope.go:117] "RemoveContainer" containerID="57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346" Oct 11 10:56:16.934994 master-1 kubenswrapper[4771]: I1011 10:56:16.934800 4771 scope.go:117] "RemoveContainer" containerID="fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53" Oct 11 10:56:16.935644 master-1 kubenswrapper[4771]: E1011 10:56:16.935407 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53\": container with ID starting with fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53 not found: ID does not exist" containerID="fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53" Oct 11 10:56:16.935808 master-1 kubenswrapper[4771]: I1011 10:56:16.935776 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53"} err="failed to get container status \"fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53\": rpc error: code = NotFound desc = could not find container \"fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53\": container with ID starting with fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53 not found: ID does not exist" Oct 11 10:56:16.935966 master-1 kubenswrapper[4771]: I1011 10:56:16.935946 4771 scope.go:117] "RemoveContainer" containerID="f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded" Oct 11 10:56:16.936988 master-1 kubenswrapper[4771]: E1011 10:56:16.936946 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded\": container with ID starting with f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded not found: ID does not exist" containerID="f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded" Oct 11 10:56:16.937090 master-1 kubenswrapper[4771]: I1011 10:56:16.936994 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded"} err="failed to get container status \"f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded\": rpc error: code = NotFound desc = could not find container \"f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded\": container with ID starting with f0b97708a382409609fd043fe1fddb88387503e83ce47d62daed66c317f5fded not found: ID does not exist" Oct 11 10:56:16.937090 master-1 kubenswrapper[4771]: I1011 10:56:16.937045 4771 scope.go:117] "RemoveContainer" containerID="57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346" Oct 11 10:56:16.937635 master-1 kubenswrapper[4771]: E1011 10:56:16.937608 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346\": container with ID starting with 57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346 not found: ID does not exist" containerID="57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346" Oct 11 10:56:16.937773 master-1 kubenswrapper[4771]: I1011 10:56:16.937744 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346"} err="failed to get container status \"57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346\": rpc error: code = NotFound desc = could not find container \"57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346\": container with ID starting with 57cea5f10f5cb919bbd746a5cc770cdb2e4c8666c371e8c10ebe7f9b79fd5346 not found: ID does not exist" Oct 11 10:56:17.788423 master-1 kubenswrapper[4771]: I1011 10:56:17.788274 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/cinder-b5802-api-0" podStartSLOduration=4.7882227969999995 podStartE2EDuration="4.788222797s" podCreationTimestamp="2025-10-11 10:56:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:56:17.62064272 +0000 UTC m=+1809.594869201" watchObservedRunningTime="2025-10-11 10:56:17.788222797 +0000 UTC m=+1809.762449278" Oct 11 10:56:17.798418 master-1 kubenswrapper[4771]: I1011 10:56:17.798318 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-4bbqs"] Oct 11 10:56:17.859642 master-1 kubenswrapper[4771]: I1011 10:56:17.859591 4771 generic.go:334] "Generic (PLEG): container finished" podID="38267a66-0ebd-44ab-bc7f-cd5703503b74" containerID="5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed" exitCode=0 Oct 11 10:56:17.859934 master-1 kubenswrapper[4771]: I1011 10:56:17.859665 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-conductor-db-sync-chcrd" event={"ID":"38267a66-0ebd-44ab-bc7f-cd5703503b74","Type":"ContainerDied","Data":"5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed"} Oct 11 10:56:17.957638 master-1 kubenswrapper[4771]: I1011 10:56:17.957583 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:17.958560 master-1 kubenswrapper[4771]: I1011 10:56:17.957737 4771 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Oct 11 10:56:17.970241 master-1 kubenswrapper[4771]: I1011 10:56:17.970153 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:56:17.971057 master-1 kubenswrapper[4771]: I1011 10:56:17.970920 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/glance-b5802-default-external-api-0" podUID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerName="glance-log" containerID="cri-o://7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474" gracePeriod=30 Oct 11 10:56:17.971795 master-1 kubenswrapper[4771]: I1011 10:56:17.971531 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/glance-b5802-default-external-api-0" podUID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerName="glance-httpd" containerID="cri-o://9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c" gracePeriod=30 Oct 11 10:56:18.045089 master-1 kubenswrapper[4771]: I1011 10:56:18.044930 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-internal-api-1" Oct 11 10:56:18.502922 master-1 kubenswrapper[4771]: I1011 10:56:18.502827 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/community-operators-4bbqs"] Oct 11 10:56:18.875612 master-1 kubenswrapper[4771]: I1011 10:56:18.875493 4771 generic.go:334] "Generic (PLEG): container finished" podID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerID="7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474" exitCode=143 Oct 11 10:56:18.876647 master-1 kubenswrapper[4771]: I1011 10:56:18.876573 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"d3028266-255a-43a3-8bdb-9695ad7cbb30","Type":"ContainerDied","Data":"7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474"} Oct 11 10:56:19.414986 master-1 kubenswrapper[4771]: I1011 10:56:19.414916 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:56:19.494841 master-1 kubenswrapper[4771]: I1011 10:56:19.494753 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-scripts\") pod \"38267a66-0ebd-44ab-bc7f-cd5703503b74\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " Oct 11 10:56:19.495161 master-1 kubenswrapper[4771]: I1011 10:56:19.494961 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-combined-ca-bundle\") pod \"38267a66-0ebd-44ab-bc7f-cd5703503b74\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " Oct 11 10:56:19.495161 master-1 kubenswrapper[4771]: I1011 10:56:19.495096 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-config-data\") pod \"38267a66-0ebd-44ab-bc7f-cd5703503b74\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " Oct 11 10:56:19.495317 master-1 kubenswrapper[4771]: I1011 10:56:19.495170 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v6lx\" (UniqueName: \"kubernetes.io/projected/38267a66-0ebd-44ab-bc7f-cd5703503b74-kube-api-access-5v6lx\") pod \"38267a66-0ebd-44ab-bc7f-cd5703503b74\" (UID: \"38267a66-0ebd-44ab-bc7f-cd5703503b74\") " Oct 11 10:56:19.499251 master-1 kubenswrapper[4771]: I1011 10:56:19.499140 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38267a66-0ebd-44ab-bc7f-cd5703503b74-kube-api-access-5v6lx" (OuterVolumeSpecName: "kube-api-access-5v6lx") pod "38267a66-0ebd-44ab-bc7f-cd5703503b74" (UID: "38267a66-0ebd-44ab-bc7f-cd5703503b74"). InnerVolumeSpecName "kube-api-access-5v6lx". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:19.516043 master-1 kubenswrapper[4771]: I1011 10:56:19.515881 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-scripts" (OuterVolumeSpecName: "scripts") pod "38267a66-0ebd-44ab-bc7f-cd5703503b74" (UID: "38267a66-0ebd-44ab-bc7f-cd5703503b74"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:19.529228 master-1 kubenswrapper[4771]: I1011 10:56:19.528973 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "38267a66-0ebd-44ab-bc7f-cd5703503b74" (UID: "38267a66-0ebd-44ab-bc7f-cd5703503b74"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:19.536328 master-1 kubenswrapper[4771]: I1011 10:56:19.536277 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-config-data" (OuterVolumeSpecName: "config-data") pod "38267a66-0ebd-44ab-bc7f-cd5703503b74" (UID: "38267a66-0ebd-44ab-bc7f-cd5703503b74"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:19.598966 master-1 kubenswrapper[4771]: I1011 10:56:19.598893 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:19.598966 master-1 kubenswrapper[4771]: I1011 10:56:19.598958 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:19.599107 master-1 kubenswrapper[4771]: I1011 10:56:19.598975 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-5v6lx\" (UniqueName: \"kubernetes.io/projected/38267a66-0ebd-44ab-bc7f-cd5703503b74-kube-api-access-5v6lx\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:19.599107 master-1 kubenswrapper[4771]: I1011 10:56:19.598991 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/38267a66-0ebd-44ab-bc7f-cd5703503b74-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:19.891446 master-1 kubenswrapper[4771]: I1011 10:56:19.891277 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-conductor-db-sync-chcrd" event={"ID":"38267a66-0ebd-44ab-bc7f-cd5703503b74","Type":"ContainerDied","Data":"0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5"} Oct 11 10:56:19.891446 master-1 kubenswrapper[4771]: I1011 10:56:19.891385 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5" Oct 11 10:56:19.892040 master-1 kubenswrapper[4771]: I1011 10:56:19.891435 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-conductor-db-sync-chcrd" Oct 11 10:56:20.449947 master-1 kubenswrapper[4771]: I1011 10:56:20.449851 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" path="/var/lib/kubelet/pods/5921e565-c581-42f4-8da8-df72fae9a3c0/volumes" Oct 11 10:56:20.728254 master-1 kubenswrapper[4771]: I1011 10:56:20.728011 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/aodh-1a24-account-create-pb6gd"] Oct 11 10:56:20.728820 master-1 kubenswrapper[4771]: E1011 10:56:20.728763 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerName="registry-server" Oct 11 10:56:20.728820 master-1 kubenswrapper[4771]: I1011 10:56:20.728808 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerName="registry-server" Oct 11 10:56:20.729026 master-1 kubenswrapper[4771]: E1011 10:56:20.728845 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerName="extract-utilities" Oct 11 10:56:20.729026 master-1 kubenswrapper[4771]: I1011 10:56:20.728862 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerName="extract-utilities" Oct 11 10:56:20.729026 master-1 kubenswrapper[4771]: E1011 10:56:20.728892 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerName="extract-content" Oct 11 10:56:20.729026 master-1 kubenswrapper[4771]: I1011 10:56:20.728908 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerName="extract-content" Oct 11 10:56:20.729026 master-1 kubenswrapper[4771]: E1011 10:56:20.728961 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="38267a66-0ebd-44ab-bc7f-cd5703503b74" containerName="nova-cell0-conductor-db-sync" Oct 11 10:56:20.729026 master-1 kubenswrapper[4771]: I1011 10:56:20.728976 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="38267a66-0ebd-44ab-bc7f-cd5703503b74" containerName="nova-cell0-conductor-db-sync" Oct 11 10:56:20.729546 master-1 kubenswrapper[4771]: I1011 10:56:20.729239 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5921e565-c581-42f4-8da8-df72fae9a3c0" containerName="registry-server" Oct 11 10:56:20.729546 master-1 kubenswrapper[4771]: I1011 10:56:20.729263 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="38267a66-0ebd-44ab-bc7f-cd5703503b74" containerName="nova-cell0-conductor-db-sync" Oct 11 10:56:20.730676 master-1 kubenswrapper[4771]: I1011 10:56:20.730624 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-1a24-account-create-pb6gd" Oct 11 10:56:20.734656 master-1 kubenswrapper[4771]: I1011 10:56:20.734589 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"aodh-db-secret" Oct 11 10:56:20.749441 master-1 kubenswrapper[4771]: I1011 10:56:20.749298 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/aodh-1a24-account-create-pb6gd"] Oct 11 10:56:20.828701 master-1 kubenswrapper[4771]: I1011 10:56:20.828625 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2t9n\" (UniqueName: \"kubernetes.io/projected/b670525b-9ca9-419c-858b-6bb2a2303cf6-kube-api-access-n2t9n\") pod \"aodh-1a24-account-create-pb6gd\" (UID: \"b670525b-9ca9-419c-858b-6bb2a2303cf6\") " pod="openstack/aodh-1a24-account-create-pb6gd" Oct 11 10:56:20.915884 master-1 kubenswrapper[4771]: I1011 10:56:20.915799 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell0-conductor-0"] Oct 11 10:56:20.922106 master-1 kubenswrapper[4771]: I1011 10:56:20.922019 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:20.927788 master-1 kubenswrapper[4771]: I1011 10:56:20.927258 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell0-conductor-config-data" Oct 11 10:56:20.951503 master-1 kubenswrapper[4771]: I1011 10:56:20.930574 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-n2t9n\" (UniqueName: \"kubernetes.io/projected/b670525b-9ca9-419c-858b-6bb2a2303cf6-kube-api-access-n2t9n\") pod \"aodh-1a24-account-create-pb6gd\" (UID: \"b670525b-9ca9-419c-858b-6bb2a2303cf6\") " pod="openstack/aodh-1a24-account-create-pb6gd" Oct 11 10:56:20.951503 master-1 kubenswrapper[4771]: I1011 10:56:20.932696 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell0-conductor-0"] Oct 11 10:56:20.960336 master-1 kubenswrapper[4771]: I1011 10:56:20.960268 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2t9n\" (UniqueName: \"kubernetes.io/projected/b670525b-9ca9-419c-858b-6bb2a2303cf6-kube-api-access-n2t9n\") pod \"aodh-1a24-account-create-pb6gd\" (UID: \"b670525b-9ca9-419c-858b-6bb2a2303cf6\") " pod="openstack/aodh-1a24-account-create-pb6gd" Oct 11 10:56:21.032765 master-1 kubenswrapper[4771]: I1011 10:56:21.032592 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c56a5472-3816-43a8-9a63-373c7893cd5c-config-data\") pod \"nova-cell0-conductor-0\" (UID: \"c56a5472-3816-43a8-9a63-373c7893cd5c\") " pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.033056 master-1 kubenswrapper[4771]: I1011 10:56:21.032839 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzwv\" (UniqueName: \"kubernetes.io/projected/c56a5472-3816-43a8-9a63-373c7893cd5c-kube-api-access-gmzwv\") pod \"nova-cell0-conductor-0\" (UID: \"c56a5472-3816-43a8-9a63-373c7893cd5c\") " pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.033056 master-1 kubenswrapper[4771]: I1011 10:56:21.032909 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c56a5472-3816-43a8-9a63-373c7893cd5c-combined-ca-bundle\") pod \"nova-cell0-conductor-0\" (UID: \"c56a5472-3816-43a8-9a63-373c7893cd5c\") " pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.078318 master-1 kubenswrapper[4771]: I1011 10:56:21.078216 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-1a24-account-create-pb6gd" Oct 11 10:56:21.135517 master-1 kubenswrapper[4771]: I1011 10:56:21.135408 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzwv\" (UniqueName: \"kubernetes.io/projected/c56a5472-3816-43a8-9a63-373c7893cd5c-kube-api-access-gmzwv\") pod \"nova-cell0-conductor-0\" (UID: \"c56a5472-3816-43a8-9a63-373c7893cd5c\") " pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.135653 master-1 kubenswrapper[4771]: I1011 10:56:21.135527 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c56a5472-3816-43a8-9a63-373c7893cd5c-combined-ca-bundle\") pod \"nova-cell0-conductor-0\" (UID: \"c56a5472-3816-43a8-9a63-373c7893cd5c\") " pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.135700 master-1 kubenswrapper[4771]: I1011 10:56:21.135673 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c56a5472-3816-43a8-9a63-373c7893cd5c-config-data\") pod \"nova-cell0-conductor-0\" (UID: \"c56a5472-3816-43a8-9a63-373c7893cd5c\") " pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.140734 master-1 kubenswrapper[4771]: I1011 10:56:21.140675 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c56a5472-3816-43a8-9a63-373c7893cd5c-combined-ca-bundle\") pod \"nova-cell0-conductor-0\" (UID: \"c56a5472-3816-43a8-9a63-373c7893cd5c\") " pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.141976 master-1 kubenswrapper[4771]: I1011 10:56:21.141923 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c56a5472-3816-43a8-9a63-373c7893cd5c-config-data\") pod \"nova-cell0-conductor-0\" (UID: \"c56a5472-3816-43a8-9a63-373c7893cd5c\") " pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.156807 master-1 kubenswrapper[4771]: I1011 10:56:21.156730 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzwv\" (UniqueName: \"kubernetes.io/projected/c56a5472-3816-43a8-9a63-373c7893cd5c-kube-api-access-gmzwv\") pod \"nova-cell0-conductor-0\" (UID: \"c56a5472-3816-43a8-9a63-373c7893cd5c\") " pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.241860 master-1 kubenswrapper[4771]: I1011 10:56:21.241800 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:21.566608 master-1 kubenswrapper[4771]: I1011 10:56:21.566554 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/aodh-1a24-account-create-pb6gd"] Oct 11 10:56:21.740974 master-1 kubenswrapper[4771]: I1011 10:56:21.740926 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell0-conductor-0"] Oct 11 10:56:21.832369 master-1 kubenswrapper[4771]: I1011 10:56:21.832312 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:21.940586 master-1 kubenswrapper[4771]: I1011 10:56:21.940484 4771 generic.go:334] "Generic (PLEG): container finished" podID="b670525b-9ca9-419c-858b-6bb2a2303cf6" containerID="34c012fefebf03c137c3d264726e9a32c974159496d4bf0d0a4dad6dcdf4c655" exitCode=0 Oct 11 10:56:21.940586 master-1 kubenswrapper[4771]: I1011 10:56:21.940598 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-1a24-account-create-pb6gd" event={"ID":"b670525b-9ca9-419c-858b-6bb2a2303cf6","Type":"ContainerDied","Data":"34c012fefebf03c137c3d264726e9a32c974159496d4bf0d0a4dad6dcdf4c655"} Oct 11 10:56:21.941524 master-1 kubenswrapper[4771]: I1011 10:56:21.940640 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-1a24-account-create-pb6gd" event={"ID":"b670525b-9ca9-419c-858b-6bb2a2303cf6","Type":"ContainerStarted","Data":"3995d8b1cb4f3f3ee48eac28241686624d899abf7b682d7cf2e0300e24841328"} Oct 11 10:56:21.943267 master-1 kubenswrapper[4771]: I1011 10:56:21.943117 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-conductor-0" event={"ID":"c56a5472-3816-43a8-9a63-373c7893cd5c","Type":"ContainerStarted","Data":"bec81004ceea111cb650b5ad6d16be3710371031464fb1a09f17b94b9ee51ee5"} Oct 11 10:56:21.946522 master-1 kubenswrapper[4771]: I1011 10:56:21.946457 4771 generic.go:334] "Generic (PLEG): container finished" podID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerID="9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c" exitCode=0 Oct 11 10:56:21.946522 master-1 kubenswrapper[4771]: I1011 10:56:21.946504 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"d3028266-255a-43a3-8bdb-9695ad7cbb30","Type":"ContainerDied","Data":"9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c"} Oct 11 10:56:21.946668 master-1 kubenswrapper[4771]: I1011 10:56:21.946521 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:21.946668 master-1 kubenswrapper[4771]: I1011 10:56:21.946542 4771 scope.go:117] "RemoveContainer" containerID="9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c" Oct 11 10:56:21.946775 master-1 kubenswrapper[4771]: I1011 10:56:21.946526 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"d3028266-255a-43a3-8bdb-9695ad7cbb30","Type":"ContainerDied","Data":"a58cd7c5986caf2dc23489be67139cc61da2a0165758a7758f40955bc3fa5183"} Oct 11 10:56:21.949718 master-1 kubenswrapper[4771]: I1011 10:56:21.949685 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-config-data\") pod \"d3028266-255a-43a3-8bdb-9695ad7cbb30\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " Oct 11 10:56:21.949801 master-1 kubenswrapper[4771]: I1011 10:56:21.949752 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-logs\") pod \"d3028266-255a-43a3-8bdb-9695ad7cbb30\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " Oct 11 10:56:21.949841 master-1 kubenswrapper[4771]: I1011 10:56:21.949804 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85jr4\" (UniqueName: \"kubernetes.io/projected/d3028266-255a-43a3-8bdb-9695ad7cbb30-kube-api-access-85jr4\") pod \"d3028266-255a-43a3-8bdb-9695ad7cbb30\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " Oct 11 10:56:21.949908 master-1 kubenswrapper[4771]: I1011 10:56:21.949875 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-httpd-run\") pod \"d3028266-255a-43a3-8bdb-9695ad7cbb30\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " Oct 11 10:56:21.949998 master-1 kubenswrapper[4771]: I1011 10:56:21.949959 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-combined-ca-bundle\") pod \"d3028266-255a-43a3-8bdb-9695ad7cbb30\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " Oct 11 10:56:21.950274 master-1 kubenswrapper[4771]: I1011 10:56:21.950247 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"glance\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"d3028266-255a-43a3-8bdb-9695ad7cbb30\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " Oct 11 10:56:21.950342 master-1 kubenswrapper[4771]: I1011 10:56:21.950318 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-public-tls-certs\") pod \"d3028266-255a-43a3-8bdb-9695ad7cbb30\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " Oct 11 10:56:21.950413 master-1 kubenswrapper[4771]: I1011 10:56:21.950338 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-logs" (OuterVolumeSpecName: "logs") pod "d3028266-255a-43a3-8bdb-9695ad7cbb30" (UID: "d3028266-255a-43a3-8bdb-9695ad7cbb30"). InnerVolumeSpecName "logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:21.950461 master-1 kubenswrapper[4771]: I1011 10:56:21.950413 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-scripts\") pod \"d3028266-255a-43a3-8bdb-9695ad7cbb30\" (UID: \"d3028266-255a-43a3-8bdb-9695ad7cbb30\") " Oct 11 10:56:21.951791 master-1 kubenswrapper[4771]: I1011 10:56:21.951465 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-httpd-run" (OuterVolumeSpecName: "httpd-run") pod "d3028266-255a-43a3-8bdb-9695ad7cbb30" (UID: "d3028266-255a-43a3-8bdb-9695ad7cbb30"). InnerVolumeSpecName "httpd-run". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:21.952567 master-1 kubenswrapper[4771]: I1011 10:56:21.952016 4771 reconciler_common.go:293] "Volume detached for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-logs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:21.952567 master-1 kubenswrapper[4771]: I1011 10:56:21.952050 4771 reconciler_common.go:293] "Volume detached for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/d3028266-255a-43a3-8bdb-9695ad7cbb30-httpd-run\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:21.955321 master-1 kubenswrapper[4771]: I1011 10:56:21.955204 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3028266-255a-43a3-8bdb-9695ad7cbb30-kube-api-access-85jr4" (OuterVolumeSpecName: "kube-api-access-85jr4") pod "d3028266-255a-43a3-8bdb-9695ad7cbb30" (UID: "d3028266-255a-43a3-8bdb-9695ad7cbb30"). InnerVolumeSpecName "kube-api-access-85jr4". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:21.958907 master-1 kubenswrapper[4771]: I1011 10:56:21.958849 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-scripts" (OuterVolumeSpecName: "scripts") pod "d3028266-255a-43a3-8bdb-9695ad7cbb30" (UID: "d3028266-255a-43a3-8bdb-9695ad7cbb30"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:21.976586 master-1 kubenswrapper[4771]: I1011 10:56:21.976414 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b" (OuterVolumeSpecName: "glance") pod "d3028266-255a-43a3-8bdb-9695ad7cbb30" (UID: "d3028266-255a-43a3-8bdb-9695ad7cbb30"). InnerVolumeSpecName "pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9". PluginName "kubernetes.io/csi", VolumeGidValue "" Oct 11 10:56:21.982024 master-1 kubenswrapper[4771]: I1011 10:56:21.981985 4771 scope.go:117] "RemoveContainer" containerID="7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474" Oct 11 10:56:21.989981 master-1 kubenswrapper[4771]: I1011 10:56:21.989892 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-config-data" (OuterVolumeSpecName: "config-data") pod "d3028266-255a-43a3-8bdb-9695ad7cbb30" (UID: "d3028266-255a-43a3-8bdb-9695ad7cbb30"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:21.999738 master-1 kubenswrapper[4771]: I1011 10:56:21.999664 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "d3028266-255a-43a3-8bdb-9695ad7cbb30" (UID: "d3028266-255a-43a3-8bdb-9695ad7cbb30"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:22.000110 master-1 kubenswrapper[4771]: I1011 10:56:22.000078 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-public-tls-certs" (OuterVolumeSpecName: "public-tls-certs") pod "d3028266-255a-43a3-8bdb-9695ad7cbb30" (UID: "d3028266-255a-43a3-8bdb-9695ad7cbb30"). InnerVolumeSpecName "public-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:22.053988 master-1 kubenswrapper[4771]: I1011 10:56:22.053926 4771 reconciler_common.go:286] "operationExecutor.UnmountDevice started for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") on node \"master-1\" " Oct 11 10:56:22.053988 master-1 kubenswrapper[4771]: I1011 10:56:22.053975 4771 reconciler_common.go:293] "Volume detached for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-public-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:22.053988 master-1 kubenswrapper[4771]: I1011 10:56:22.053990 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:22.053988 master-1 kubenswrapper[4771]: I1011 10:56:22.054000 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:22.053988 master-1 kubenswrapper[4771]: I1011 10:56:22.054014 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-85jr4\" (UniqueName: \"kubernetes.io/projected/d3028266-255a-43a3-8bdb-9695ad7cbb30-kube-api-access-85jr4\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:22.054279 master-1 kubenswrapper[4771]: I1011 10:56:22.054029 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3028266-255a-43a3-8bdb-9695ad7cbb30-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:22.077056 master-1 kubenswrapper[4771]: I1011 10:56:22.077003 4771 scope.go:117] "RemoveContainer" containerID="9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c" Oct 11 10:56:22.077908 master-1 kubenswrapper[4771]: E1011 10:56:22.077872 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c\": container with ID starting with 9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c not found: ID does not exist" containerID="9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c" Oct 11 10:56:22.077987 master-1 kubenswrapper[4771]: I1011 10:56:22.077916 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c"} err="failed to get container status \"9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c\": rpc error: code = NotFound desc = could not find container \"9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c\": container with ID starting with 9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c not found: ID does not exist" Oct 11 10:56:22.077987 master-1 kubenswrapper[4771]: I1011 10:56:22.077945 4771 scope.go:117] "RemoveContainer" containerID="7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474" Oct 11 10:56:22.078737 master-1 kubenswrapper[4771]: E1011 10:56:22.078626 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474\": container with ID starting with 7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474 not found: ID does not exist" containerID="7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474" Oct 11 10:56:22.078737 master-1 kubenswrapper[4771]: I1011 10:56:22.078698 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474"} err="failed to get container status \"7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474\": rpc error: code = NotFound desc = could not find container \"7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474\": container with ID starting with 7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474 not found: ID does not exist" Oct 11 10:56:22.081085 master-1 kubenswrapper[4771]: I1011 10:56:22.081052 4771 csi_attacher.go:630] kubernetes.io/csi: attacher.UnmountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping UnmountDevice... Oct 11 10:56:22.081272 master-1 kubenswrapper[4771]: I1011 10:56:22.081251 4771 operation_generator.go:917] UnmountDevice succeeded for volume "pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9" (UniqueName: "kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b") on node "master-1" Oct 11 10:56:22.157608 master-1 kubenswrapper[4771]: I1011 10:56:22.157531 4771 reconciler_common.go:293] "Volume detached for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:22.312757 master-1 kubenswrapper[4771]: I1011 10:56:22.312686 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:56:22.322299 master-1 kubenswrapper[4771]: I1011 10:56:22.322218 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:56:22.354402 master-1 kubenswrapper[4771]: I1011 10:56:22.354307 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:56:22.354858 master-1 kubenswrapper[4771]: E1011 10:56:22.354823 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerName="glance-httpd" Oct 11 10:56:22.354944 master-1 kubenswrapper[4771]: I1011 10:56:22.354856 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerName="glance-httpd" Oct 11 10:56:22.354944 master-1 kubenswrapper[4771]: E1011 10:56:22.354895 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerName="glance-log" Oct 11 10:56:22.354944 master-1 kubenswrapper[4771]: I1011 10:56:22.354910 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerName="glance-log" Oct 11 10:56:22.355196 master-1 kubenswrapper[4771]: I1011 10:56:22.355167 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerName="glance-log" Oct 11 10:56:22.355248 master-1 kubenswrapper[4771]: I1011 10:56:22.355231 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d3028266-255a-43a3-8bdb-9695ad7cbb30" containerName="glance-httpd" Oct 11 10:56:22.357162 master-1 kubenswrapper[4771]: I1011 10:56:22.357132 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.360801 master-1 kubenswrapper[4771]: I1011 10:56:22.360777 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"glance-b5802-default-external-config-data" Oct 11 10:56:22.360988 master-1 kubenswrapper[4771]: I1011 10:56:22.360950 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-glance-default-public-svc" Oct 11 10:56:22.384496 master-1 kubenswrapper[4771]: I1011 10:56:22.384418 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:56:22.505840 master-1 kubenswrapper[4771]: I1011 10:56:22.504532 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-config-data\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.505840 master-1 kubenswrapper[4771]: I1011 10:56:22.505073 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.505840 master-1 kubenswrapper[4771]: I1011 10:56:22.505205 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/d8e3e70e-31b7-4245-b138-fdc9401dd344-logs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.505840 master-1 kubenswrapper[4771]: I1011 10:56:22.505440 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwf9w\" (UniqueName: \"kubernetes.io/projected/d8e3e70e-31b7-4245-b138-fdc9401dd344-kube-api-access-vwf9w\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.505840 master-1 kubenswrapper[4771]: I1011 10:56:22.505604 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-scripts\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.506182 master-1 kubenswrapper[4771]: I1011 10:56:22.505873 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/d8e3e70e-31b7-4245-b138-fdc9401dd344-httpd-run\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.506182 master-1 kubenswrapper[4771]: I1011 10:56:22.505973 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-combined-ca-bundle\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.506182 master-1 kubenswrapper[4771]: I1011 10:56:22.506095 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-public-tls-certs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.507143 master-1 kubenswrapper[4771]: I1011 10:56:22.507069 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3028266-255a-43a3-8bdb-9695ad7cbb30" path="/var/lib/kubelet/pods/d3028266-255a-43a3-8bdb-9695ad7cbb30/volumes" Oct 11 10:56:22.608101 master-1 kubenswrapper[4771]: I1011 10:56:22.608016 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-scripts\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.608498 master-1 kubenswrapper[4771]: I1011 10:56:22.608156 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/d8e3e70e-31b7-4245-b138-fdc9401dd344-httpd-run\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.608498 master-1 kubenswrapper[4771]: I1011 10:56:22.608296 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-combined-ca-bundle\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.608498 master-1 kubenswrapper[4771]: I1011 10:56:22.608348 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-public-tls-certs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.608741 master-1 kubenswrapper[4771]: I1011 10:56:22.608711 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-config-data\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.608813 master-1 kubenswrapper[4771]: I1011 10:56:22.608794 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.608885 master-1 kubenswrapper[4771]: I1011 10:56:22.608838 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/d8e3e70e-31b7-4245-b138-fdc9401dd344-logs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.608976 master-1 kubenswrapper[4771]: I1011 10:56:22.608882 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-vwf9w\" (UniqueName: \"kubernetes.io/projected/d8e3e70e-31b7-4245-b138-fdc9401dd344-kube-api-access-vwf9w\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.609794 master-1 kubenswrapper[4771]: I1011 10:56:22.609683 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-run\" (UniqueName: \"kubernetes.io/empty-dir/d8e3e70e-31b7-4245-b138-fdc9401dd344-httpd-run\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.609950 master-1 kubenswrapper[4771]: I1011 10:56:22.609800 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/d8e3e70e-31b7-4245-b138-fdc9401dd344-logs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.613435 master-1 kubenswrapper[4771]: I1011 10:56:22.613396 4771 csi_attacher.go:380] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice... Oct 11 10:56:22.613435 master-1 kubenswrapper[4771]: I1011 10:56:22.613435 4771 operation_generator.go:580] "MountVolume.MountDevice succeeded for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/topolvm.io/643ba808821ea6db76a2042d255ba68bbc43444ed3cc7e332598424f5540da0c/globalmount\"" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.614278 master-1 kubenswrapper[4771]: I1011 10:56:22.614192 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-combined-ca-bundle\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.615074 master-1 kubenswrapper[4771]: I1011 10:56:22.615016 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-scripts\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.615185 master-1 kubenswrapper[4771]: I1011 10:56:22.615039 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-public-tls-certs\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.616595 master-1 kubenswrapper[4771]: I1011 10:56:22.616544 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d8e3e70e-31b7-4245-b138-fdc9401dd344-config-data\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.635598 master-1 kubenswrapper[4771]: I1011 10:56:22.635529 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwf9w\" (UniqueName: \"kubernetes.io/projected/d8e3e70e-31b7-4245-b138-fdc9401dd344-kube-api-access-vwf9w\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:22.966900 master-1 kubenswrapper[4771]: I1011 10:56:22.966837 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-conductor-0" event={"ID":"c56a5472-3816-43a8-9a63-373c7893cd5c","Type":"ContainerStarted","Data":"2f137269af75815653291ff48643748db0ace9751acbd8894e3285de0f3bbbeb"} Oct 11 10:56:22.967647 master-1 kubenswrapper[4771]: I1011 10:56:22.966930 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:23.000754 master-1 kubenswrapper[4771]: I1011 10:56:23.000646 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell0-conductor-0" podStartSLOduration=3.000626686 podStartE2EDuration="3.000626686s" podCreationTimestamp="2025-10-11 10:56:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:56:22.995402114 +0000 UTC m=+1814.969628605" watchObservedRunningTime="2025-10-11 10:56:23.000626686 +0000 UTC m=+1814.974853137" Oct 11 10:56:23.421371 master-1 kubenswrapper[4771]: I1011 10:56:23.420678 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-1a24-account-create-pb6gd" Oct 11 10:56:23.527264 master-1 kubenswrapper[4771]: I1011 10:56:23.527195 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"pvc-61680409-2444-4e5f-9b6b-1cb1b48ecfb9\" (UniqueName: \"kubernetes.io/csi/topolvm.io^e0cfc426-71d8-44a8-9d4b-7bb8001fb12b\") pod \"glance-b5802-default-external-api-0\" (UID: \"d8e3e70e-31b7-4245-b138-fdc9401dd344\") " pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:23.549179 master-1 kubenswrapper[4771]: I1011 10:56:23.549111 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2t9n\" (UniqueName: \"kubernetes.io/projected/b670525b-9ca9-419c-858b-6bb2a2303cf6-kube-api-access-n2t9n\") pod \"b670525b-9ca9-419c-858b-6bb2a2303cf6\" (UID: \"b670525b-9ca9-419c-858b-6bb2a2303cf6\") " Oct 11 10:56:23.552338 master-1 kubenswrapper[4771]: I1011 10:56:23.552295 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b670525b-9ca9-419c-858b-6bb2a2303cf6-kube-api-access-n2t9n" (OuterVolumeSpecName: "kube-api-access-n2t9n") pod "b670525b-9ca9-419c-858b-6bb2a2303cf6" (UID: "b670525b-9ca9-419c-858b-6bb2a2303cf6"). InnerVolumeSpecName "kube-api-access-n2t9n". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:23.583775 master-1 kubenswrapper[4771]: I1011 10:56:23.583716 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:23.652545 master-1 kubenswrapper[4771]: I1011 10:56:23.652352 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-n2t9n\" (UniqueName: \"kubernetes.io/projected/b670525b-9ca9-419c-858b-6bb2a2303cf6-kube-api-access-n2t9n\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:23.976813 master-1 kubenswrapper[4771]: I1011 10:56:23.976595 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-1a24-account-create-pb6gd" event={"ID":"b670525b-9ca9-419c-858b-6bb2a2303cf6","Type":"ContainerDied","Data":"3995d8b1cb4f3f3ee48eac28241686624d899abf7b682d7cf2e0300e24841328"} Oct 11 10:56:23.976813 master-1 kubenswrapper[4771]: I1011 10:56:23.976682 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3995d8b1cb4f3f3ee48eac28241686624d899abf7b682d7cf2e0300e24841328" Oct 11 10:56:23.976813 master-1 kubenswrapper[4771]: I1011 10:56:23.976620 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-1a24-account-create-pb6gd" Oct 11 10:56:24.005618 master-1 kubenswrapper[4771]: E1011 10:56:24.005555 4771 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/bdc960f594c7cf7b61069e0e0abb31efcf62303e52388cd0c49a5755236d9f74/diff" to get inode usage: stat /var/lib/containers/storage/overlay/bdc960f594c7cf7b61069e0e0abb31efcf62303e52388cd0c49a5755236d9f74/diff: no such file or directory, extraDiskErr: Oct 11 10:56:24.168799 master-1 kubenswrapper[4771]: I1011 10:56:24.164759 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/glance-b5802-default-external-api-0"] Oct 11 10:56:24.295790 master-1 kubenswrapper[4771]: E1011 10:56:24.295608 4771 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/bfd5c1f52f3f50fe03a7dc129b7fce9a308b01a368798b89ac9e8ed524a70b73/diff" to get inode usage: stat /var/lib/containers/storage/overlay/bfd5c1f52f3f50fe03a7dc129b7fce9a308b01a368798b89ac9e8ed524a70b73/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openstack_cinder-b5802-api-0_478147ef-a0d7-4c37-952c-3fc3a23775db/cinder-api/0.log" to get inode usage: stat /var/log/pods/openstack_cinder-b5802-api-0_478147ef-a0d7-4c37-952c-3fc3a23775db/cinder-api/0.log: no such file or directory Oct 11 10:56:24.710875 master-1 kubenswrapper[4771]: E1011 10:56:24.708719 4771 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/52e44d3adbc247b2634220b9a00f35ef026c5019f60eab176f475b3a36c5dd65/diff" to get inode usage: stat /var/lib/containers/storage/overlay/52e44d3adbc247b2634220b9a00f35ef026c5019f60eab176f475b3a36c5dd65/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openstack_glance-b5802-default-external-api-0_d3028266-255a-43a3-8bdb-9695ad7cbb30/glance-log/0.log" to get inode usage: stat /var/log/pods/openstack_glance-b5802-default-external-api-0_d3028266-255a-43a3-8bdb-9695ad7cbb30/glance-log/0.log: no such file or directory Oct 11 10:56:24.994019 master-1 kubenswrapper[4771]: I1011 10:56:24.993825 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"d8e3e70e-31b7-4245-b138-fdc9401dd344","Type":"ContainerStarted","Data":"0e44421f3bf9e59656d4af212a8941201c6c81fac80f9388ebb84f64d53b59b9"} Oct 11 10:56:24.994019 master-1 kubenswrapper[4771]: I1011 10:56:24.993912 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"d8e3e70e-31b7-4245-b138-fdc9401dd344","Type":"ContainerStarted","Data":"fa33ed8483db0624db533dce5d7a3e460dd7d608c7626bc3e38403fe08260f0e"} Oct 11 10:56:25.607509 master-1 kubenswrapper[4771]: E1011 10:56:25.607441 4771 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/a339af6a375a54fe7e2efcfe927dc406a1de8e87afa50ea2a74db4db8b3b419d/diff" to get inode usage: stat /var/lib/containers/storage/overlay/a339af6a375a54fe7e2efcfe927dc406a1de8e87afa50ea2a74db4db8b3b419d/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openstack_glance-b5802-default-external-api-0_d3028266-255a-43a3-8bdb-9695ad7cbb30/glance-httpd/0.log" to get inode usage: stat /var/log/pods/openstack_glance-b5802-default-external-api-0_d3028266-255a-43a3-8bdb-9695ad7cbb30/glance-httpd/0.log: no such file or directory Oct 11 10:56:26.011838 master-1 kubenswrapper[4771]: I1011 10:56:26.011696 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/glance-b5802-default-external-api-0" event={"ID":"d8e3e70e-31b7-4245-b138-fdc9401dd344","Type":"ContainerStarted","Data":"1ddd60e696582c7a2790a2a426954d0b37bf6881aab474d30558dc2354cbccd2"} Oct 11 10:56:26.059492 master-1 kubenswrapper[4771]: I1011 10:56:26.059424 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/glance-b5802-default-external-api-0" podStartSLOduration=4.059400988 podStartE2EDuration="4.059400988s" podCreationTimestamp="2025-10-11 10:56:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:56:26.05159207 +0000 UTC m=+1818.025818531" watchObservedRunningTime="2025-10-11 10:56:26.059400988 +0000 UTC m=+1818.033627429" Oct 11 10:56:26.116747 master-1 kubenswrapper[4771]: I1011 10:56:26.116674 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/aodh-db-sync-tn8xz"] Oct 11 10:56:26.117542 master-1 kubenswrapper[4771]: E1011 10:56:26.117521 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="b670525b-9ca9-419c-858b-6bb2a2303cf6" containerName="mariadb-account-create" Oct 11 10:56:26.117642 master-1 kubenswrapper[4771]: I1011 10:56:26.117630 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="b670525b-9ca9-419c-858b-6bb2a2303cf6" containerName="mariadb-account-create" Oct 11 10:56:26.117893 master-1 kubenswrapper[4771]: I1011 10:56:26.117876 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="b670525b-9ca9-419c-858b-6bb2a2303cf6" containerName="mariadb-account-create" Oct 11 10:56:26.118882 master-1 kubenswrapper[4771]: I1011 10:56:26.118861 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.123936 master-1 kubenswrapper[4771]: I1011 10:56:26.123828 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"aodh-config-data" Oct 11 10:56:26.124085 master-1 kubenswrapper[4771]: I1011 10:56:26.123839 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"aodh-scripts" Oct 11 10:56:26.129272 master-1 kubenswrapper[4771]: I1011 10:56:26.129220 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/aodh-db-sync-tn8xz"] Oct 11 10:56:26.214332 master-1 kubenswrapper[4771]: I1011 10:56:26.214270 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mgq2\" (UniqueName: \"kubernetes.io/projected/3de492fb-5249-49e2-a327-756234aa92bd-kube-api-access-6mgq2\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.214756 master-1 kubenswrapper[4771]: I1011 10:56:26.214402 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-scripts\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.214756 master-1 kubenswrapper[4771]: I1011 10:56:26.214463 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-config-data\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.214756 master-1 kubenswrapper[4771]: I1011 10:56:26.214557 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-combined-ca-bundle\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.270019 master-1 kubenswrapper[4771]: I1011 10:56:26.269870 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-cell0-conductor-0" Oct 11 10:56:26.316304 master-1 kubenswrapper[4771]: I1011 10:56:26.316246 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-6mgq2\" (UniqueName: \"kubernetes.io/projected/3de492fb-5249-49e2-a327-756234aa92bd-kube-api-access-6mgq2\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.316548 master-1 kubenswrapper[4771]: I1011 10:56:26.316327 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-scripts\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.316548 master-1 kubenswrapper[4771]: I1011 10:56:26.316389 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-config-data\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.316548 master-1 kubenswrapper[4771]: I1011 10:56:26.316442 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-combined-ca-bundle\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.320690 master-1 kubenswrapper[4771]: I1011 10:56:26.320647 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-combined-ca-bundle\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.320921 master-1 kubenswrapper[4771]: I1011 10:56:26.320891 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-config-data\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.332644 master-1 kubenswrapper[4771]: I1011 10:56:26.331460 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/cinder-b5802-api-0" Oct 11 10:56:26.333696 master-1 kubenswrapper[4771]: I1011 10:56:26.333475 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-scripts\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.336777 master-1 kubenswrapper[4771]: I1011 10:56:26.336740 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mgq2\" (UniqueName: \"kubernetes.io/projected/3de492fb-5249-49e2-a327-756234aa92bd-kube-api-access-6mgq2\") pod \"aodh-db-sync-tn8xz\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.447323 master-1 kubenswrapper[4771]: I1011 10:56:26.447292 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:26.896600 master-1 kubenswrapper[4771]: I1011 10:56:26.896544 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/aodh-db-sync-tn8xz"] Oct 11 10:56:26.909370 master-1 kubenswrapper[4771]: I1011 10:56:26.909305 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell0-cell-mapping-2kt7k"] Oct 11 10:56:26.911006 master-1 kubenswrapper[4771]: I1011 10:56:26.910964 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:26.914876 master-1 kubenswrapper[4771]: I1011 10:56:26.914812 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell0-manage-scripts" Oct 11 10:56:26.914969 master-1 kubenswrapper[4771]: I1011 10:56:26.914848 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell0-manage-config-data" Oct 11 10:56:26.933808 master-1 kubenswrapper[4771]: I1011 10:56:26.933730 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell0-cell-mapping-2kt7k"] Oct 11 10:56:27.033389 master-1 kubenswrapper[4771]: I1011 10:56:27.032759 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-db-sync-tn8xz" event={"ID":"3de492fb-5249-49e2-a327-756234aa92bd","Type":"ContainerStarted","Data":"4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344"} Oct 11 10:56:27.044149 master-1 kubenswrapper[4771]: I1011 10:56:27.044069 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell1-compute-ironic-compute-0"] Oct 11 10:56:27.047130 master-1 kubenswrapper[4771]: I1011 10:56:27.047061 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.051978 master-1 kubenswrapper[4771]: I1011 10:56:27.051920 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell1-compute-ironic-compute-config-data" Oct 11 10:56:27.066809 master-1 kubenswrapper[4771]: I1011 10:56:27.063218 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-compute-ironic-compute-0"] Oct 11 10:56:27.080343 master-1 kubenswrapper[4771]: I1011 10:56:27.076671 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clg6t\" (UniqueName: \"kubernetes.io/projected/f85d5cfa-8073-4bbf-9eff-78fde719dadf-kube-api-access-clg6t\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.080343 master-1 kubenswrapper[4771]: I1011 10:56:27.076965 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-scripts\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.080343 master-1 kubenswrapper[4771]: I1011 10:56:27.077136 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-combined-ca-bundle\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.080343 master-1 kubenswrapper[4771]: I1011 10:56:27.077224 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-config-data\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.179431 master-1 kubenswrapper[4771]: I1011 10:56:27.179347 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6bt5\" (UniqueName: \"kubernetes.io/projected/3a8c3af6-0b6a-486d-83e3-18bf00346dbc-kube-api-access-c6bt5\") pod \"nova-cell1-compute-ironic-compute-0\" (UID: \"3a8c3af6-0b6a-486d-83e3-18bf00346dbc\") " pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.179617 master-1 kubenswrapper[4771]: I1011 10:56:27.179502 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-scripts\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.179656 master-1 kubenswrapper[4771]: I1011 10:56:27.179591 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/3a8c3af6-0b6a-486d-83e3-18bf00346dbc-config-data\") pod \"nova-cell1-compute-ironic-compute-0\" (UID: \"3a8c3af6-0b6a-486d-83e3-18bf00346dbc\") " pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.179843 master-1 kubenswrapper[4771]: I1011 10:56:27.179806 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-combined-ca-bundle\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.179898 master-1 kubenswrapper[4771]: I1011 10:56:27.179863 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3a8c3af6-0b6a-486d-83e3-18bf00346dbc-combined-ca-bundle\") pod \"nova-cell1-compute-ironic-compute-0\" (UID: \"3a8c3af6-0b6a-486d-83e3-18bf00346dbc\") " pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.182430 master-1 kubenswrapper[4771]: I1011 10:56:27.179989 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-config-data\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.182430 master-1 kubenswrapper[4771]: I1011 10:56:27.180212 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-clg6t\" (UniqueName: \"kubernetes.io/projected/f85d5cfa-8073-4bbf-9eff-78fde719dadf-kube-api-access-clg6t\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.186226 master-1 kubenswrapper[4771]: I1011 10:56:27.186168 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-combined-ca-bundle\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.187020 master-1 kubenswrapper[4771]: I1011 10:56:27.186968 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-scripts\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.189175 master-1 kubenswrapper[4771]: I1011 10:56:27.189116 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-config-data\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.212399 master-1 kubenswrapper[4771]: I1011 10:56:27.212309 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-clg6t\" (UniqueName: \"kubernetes.io/projected/f85d5cfa-8073-4bbf-9eff-78fde719dadf-kube-api-access-clg6t\") pod \"nova-cell0-cell-mapping-2kt7k\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.213034 master-1 kubenswrapper[4771]: I1011 10:56:27.212980 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-api-0"] Oct 11 10:56:27.219781 master-1 kubenswrapper[4771]: I1011 10:56:27.219111 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-0" Oct 11 10:56:27.225403 master-1 kubenswrapper[4771]: I1011 10:56:27.225342 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-api-config-data" Oct 11 10:56:27.230236 master-1 kubenswrapper[4771]: I1011 10:56:27.230152 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-api-0"] Oct 11 10:56:27.256578 master-1 kubenswrapper[4771]: I1011 10:56:27.256512 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:56:27.259426 master-1 kubenswrapper[4771]: I1011 10:56:27.259385 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:27.261567 master-1 kubenswrapper[4771]: I1011 10:56:27.261538 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:56:27.264340 master-1 kubenswrapper[4771]: I1011 10:56:27.264300 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:56:27.265758 master-1 kubenswrapper[4771]: I1011 10:56:27.265720 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-scheduler-config-data" Oct 11 10:56:27.284076 master-1 kubenswrapper[4771]: I1011 10:56:27.282578 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-c6bt5\" (UniqueName: \"kubernetes.io/projected/3a8c3af6-0b6a-486d-83e3-18bf00346dbc-kube-api-access-c6bt5\") pod \"nova-cell1-compute-ironic-compute-0\" (UID: \"3a8c3af6-0b6a-486d-83e3-18bf00346dbc\") " pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.284347 master-1 kubenswrapper[4771]: I1011 10:56:27.284307 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/3a8c3af6-0b6a-486d-83e3-18bf00346dbc-config-data\") pod \"nova-cell1-compute-ironic-compute-0\" (UID: \"3a8c3af6-0b6a-486d-83e3-18bf00346dbc\") " pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.284597 master-1 kubenswrapper[4771]: I1011 10:56:27.284575 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3a8c3af6-0b6a-486d-83e3-18bf00346dbc-combined-ca-bundle\") pod \"nova-cell1-compute-ironic-compute-0\" (UID: \"3a8c3af6-0b6a-486d-83e3-18bf00346dbc\") " pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.290837 master-1 kubenswrapper[4771]: I1011 10:56:27.287392 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/3a8c3af6-0b6a-486d-83e3-18bf00346dbc-config-data\") pod \"nova-cell1-compute-ironic-compute-0\" (UID: \"3a8c3af6-0b6a-486d-83e3-18bf00346dbc\") " pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.292645 master-1 kubenswrapper[4771]: I1011 10:56:27.292607 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3a8c3af6-0b6a-486d-83e3-18bf00346dbc-combined-ca-bundle\") pod \"nova-cell1-compute-ironic-compute-0\" (UID: \"3a8c3af6-0b6a-486d-83e3-18bf00346dbc\") " pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.309261 master-1 kubenswrapper[4771]: I1011 10:56:27.309210 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6bt5\" (UniqueName: \"kubernetes.io/projected/3a8c3af6-0b6a-486d-83e3-18bf00346dbc-kube-api-access-c6bt5\") pod \"nova-cell1-compute-ironic-compute-0\" (UID: \"3a8c3af6-0b6a-486d-83e3-18bf00346dbc\") " pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.351931 master-1 kubenswrapper[4771]: I1011 10:56:27.351106 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell1-novncproxy-0"] Oct 11 10:56:27.356779 master-1 kubenswrapper[4771]: I1011 10:56:27.353898 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.357659 master-1 kubenswrapper[4771]: I1011 10:56:27.357622 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell1-novncproxy-config-data" Oct 11 10:56:27.366053 master-1 kubenswrapper[4771]: I1011 10:56:27.366016 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-novncproxy-0"] Oct 11 10:56:27.381042 master-1 kubenswrapper[4771]: I1011 10:56:27.379565 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:27.393431 master-1 kubenswrapper[4771]: I1011 10:56:27.392904 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/c648855c-73f8-4316-9eca-147821b776c2-logs\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.393431 master-1 kubenswrapper[4771]: I1011 10:56:27.392969 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-config-data\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.393431 master-1 kubenswrapper[4771]: I1011 10:56:27.393039 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-combined-ca-bundle\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.393431 master-1 kubenswrapper[4771]: I1011 10:56:27.393069 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-config-data\") pod \"nova-scheduler-1\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " pod="openstack/nova-scheduler-1" Oct 11 10:56:27.393431 master-1 kubenswrapper[4771]: I1011 10:56:27.393117 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjhk\" (UniqueName: \"kubernetes.io/projected/4619bcb1-090e-4824-adfe-6a526158d0ea-kube-api-access-srjhk\") pod \"nova-scheduler-1\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " pod="openstack/nova-scheduler-1" Oct 11 10:56:27.393431 master-1 kubenswrapper[4771]: I1011 10:56:27.393162 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-combined-ca-bundle\") pod \"nova-scheduler-1\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " pod="openstack/nova-scheduler-1" Oct 11 10:56:27.393431 master-1 kubenswrapper[4771]: I1011 10:56:27.393196 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcdx\" (UniqueName: \"kubernetes.io/projected/c648855c-73f8-4316-9eca-147821b776c2-kube-api-access-rkcdx\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.495542 master-1 kubenswrapper[4771]: I1011 10:56:27.495221 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-combined-ca-bundle\") pod \"nova-cell1-novncproxy-0\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.495542 master-1 kubenswrapper[4771]: I1011 10:56:27.495321 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/c648855c-73f8-4316-9eca-147821b776c2-logs\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.495542 master-1 kubenswrapper[4771]: I1011 10:56:27.495385 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-config-data\") pod \"nova-cell1-novncproxy-0\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.495542 master-1 kubenswrapper[4771]: I1011 10:56:27.495426 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-config-data\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.496279 master-1 kubenswrapper[4771]: I1011 10:56:27.495488 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5zqd\" (UniqueName: \"kubernetes.io/projected/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-kube-api-access-g5zqd\") pod \"nova-cell1-novncproxy-0\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.496279 master-1 kubenswrapper[4771]: I1011 10:56:27.495987 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-combined-ca-bundle\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.496279 master-1 kubenswrapper[4771]: I1011 10:56:27.496025 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-config-data\") pod \"nova-scheduler-1\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " pod="openstack/nova-scheduler-1" Oct 11 10:56:27.496279 master-1 kubenswrapper[4771]: I1011 10:56:27.496077 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-srjhk\" (UniqueName: \"kubernetes.io/projected/4619bcb1-090e-4824-adfe-6a526158d0ea-kube-api-access-srjhk\") pod \"nova-scheduler-1\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " pod="openstack/nova-scheduler-1" Oct 11 10:56:27.496279 master-1 kubenswrapper[4771]: I1011 10:56:27.496118 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-combined-ca-bundle\") pod \"nova-scheduler-1\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " pod="openstack/nova-scheduler-1" Oct 11 10:56:27.496279 master-1 kubenswrapper[4771]: I1011 10:56:27.496151 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcdx\" (UniqueName: \"kubernetes.io/projected/c648855c-73f8-4316-9eca-147821b776c2-kube-api-access-rkcdx\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.497158 master-1 kubenswrapper[4771]: I1011 10:56:27.497099 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/c648855c-73f8-4316-9eca-147821b776c2-logs\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.500051 master-1 kubenswrapper[4771]: I1011 10:56:27.500003 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-config-data\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.500519 master-1 kubenswrapper[4771]: I1011 10:56:27.500493 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-combined-ca-bundle\") pod \"nova-scheduler-1\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " pod="openstack/nova-scheduler-1" Oct 11 10:56:27.502290 master-1 kubenswrapper[4771]: I1011 10:56:27.502248 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-combined-ca-bundle\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.510885 master-1 kubenswrapper[4771]: I1011 10:56:27.508427 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-config-data\") pod \"nova-scheduler-1\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " pod="openstack/nova-scheduler-1" Oct 11 10:56:27.521106 master-1 kubenswrapper[4771]: I1011 10:56:27.521069 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcdx\" (UniqueName: \"kubernetes.io/projected/c648855c-73f8-4316-9eca-147821b776c2-kube-api-access-rkcdx\") pod \"nova-api-0\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " pod="openstack/nova-api-0" Oct 11 10:56:27.543342 master-1 kubenswrapper[4771]: I1011 10:56:27.543276 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjhk\" (UniqueName: \"kubernetes.io/projected/4619bcb1-090e-4824-adfe-6a526158d0ea-kube-api-access-srjhk\") pod \"nova-scheduler-1\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " pod="openstack/nova-scheduler-1" Oct 11 10:56:27.549205 master-1 kubenswrapper[4771]: I1011 10:56:27.549152 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-0" Oct 11 10:56:27.580344 master-1 kubenswrapper[4771]: I1011 10:56:27.580229 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:56:27.599287 master-1 kubenswrapper[4771]: I1011 10:56:27.599234 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zqd\" (UniqueName: \"kubernetes.io/projected/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-kube-api-access-g5zqd\") pod \"nova-cell1-novncproxy-0\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.599594 master-1 kubenswrapper[4771]: I1011 10:56:27.599400 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-combined-ca-bundle\") pod \"nova-cell1-novncproxy-0\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.599973 master-1 kubenswrapper[4771]: I1011 10:56:27.599787 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-config-data\") pod \"nova-cell1-novncproxy-0\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.604128 master-1 kubenswrapper[4771]: I1011 10:56:27.604078 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-combined-ca-bundle\") pod \"nova-cell1-novncproxy-0\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.606381 master-1 kubenswrapper[4771]: I1011 10:56:27.606331 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-config-data\") pod \"nova-cell1-novncproxy-0\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.623254 master-1 kubenswrapper[4771]: I1011 10:56:27.623213 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:56:27.628251 master-1 kubenswrapper[4771]: I1011 10:56:27.625038 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:56:27.629675 master-1 kubenswrapper[4771]: I1011 10:56:27.629619 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-metadata-config-data" Oct 11 10:56:27.631413 master-1 kubenswrapper[4771]: I1011 10:56:27.630834 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zqd\" (UniqueName: \"kubernetes.io/projected/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-kube-api-access-g5zqd\") pod \"nova-cell1-novncproxy-0\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.636173 master-1 kubenswrapper[4771]: I1011 10:56:27.636111 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:56:27.728456 master-1 kubenswrapper[4771]: I1011 10:56:27.728297 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell0-cell-mapping-2kt7k"] Oct 11 10:56:27.776764 master-1 kubenswrapper[4771]: I1011 10:56:27.776714 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:27.806752 master-1 kubenswrapper[4771]: I1011 10:56:27.806691 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-combined-ca-bundle\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.806953 master-1 kubenswrapper[4771]: I1011 10:56:27.806760 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/1596746b-25ca-487a-9e49-93e532f2838b-logs\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.806953 master-1 kubenswrapper[4771]: I1011 10:56:27.806802 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scqmr\" (UniqueName: \"kubernetes.io/projected/1596746b-25ca-487a-9e49-93e532f2838b-kube-api-access-scqmr\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.806953 master-1 kubenswrapper[4771]: I1011 10:56:27.806946 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-config-data\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.873496 master-1 kubenswrapper[4771]: I1011 10:56:27.873415 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-compute-ironic-compute-0"] Oct 11 10:56:27.909221 master-1 kubenswrapper[4771]: I1011 10:56:27.909162 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-combined-ca-bundle\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.909491 master-1 kubenswrapper[4771]: I1011 10:56:27.909240 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/1596746b-25ca-487a-9e49-93e532f2838b-logs\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.909491 master-1 kubenswrapper[4771]: I1011 10:56:27.909310 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-scqmr\" (UniqueName: \"kubernetes.io/projected/1596746b-25ca-487a-9e49-93e532f2838b-kube-api-access-scqmr\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.909491 master-1 kubenswrapper[4771]: I1011 10:56:27.909445 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-config-data\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.909956 master-1 kubenswrapper[4771]: I1011 10:56:27.909910 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/1596746b-25ca-487a-9e49-93e532f2838b-logs\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.914179 master-1 kubenswrapper[4771]: I1011 10:56:27.914121 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-combined-ca-bundle\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.916863 master-1 kubenswrapper[4771]: I1011 10:56:27.916679 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-config-data\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.940425 master-1 kubenswrapper[4771]: I1011 10:56:27.937530 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-scqmr\" (UniqueName: \"kubernetes.io/projected/1596746b-25ca-487a-9e49-93e532f2838b-kube-api-access-scqmr\") pod \"nova-metadata-1\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " pod="openstack/nova-metadata-1" Oct 11 10:56:27.961521 master-1 kubenswrapper[4771]: I1011 10:56:27.961464 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:56:28.044878 master-1 kubenswrapper[4771]: I1011 10:56:28.044814 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-compute-ironic-compute-0" event={"ID":"3a8c3af6-0b6a-486d-83e3-18bf00346dbc","Type":"ContainerStarted","Data":"21b5136e598560355383aa18f08807d7668d8be22b7a488b7e02d1b5439e130e"} Oct 11 10:56:28.063256 master-1 kubenswrapper[4771]: I1011 10:56:28.052434 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-cell-mapping-2kt7k" event={"ID":"f85d5cfa-8073-4bbf-9eff-78fde719dadf","Type":"ContainerStarted","Data":"03e80f5bdb6844a3112427ed3612b145765c86f689c582771359401e14c9758e"} Oct 11 10:56:28.063256 master-1 kubenswrapper[4771]: I1011 10:56:28.052512 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-cell-mapping-2kt7k" event={"ID":"f85d5cfa-8073-4bbf-9eff-78fde719dadf","Type":"ContainerStarted","Data":"792c6e93eebb0025120939bb299c7a87876ec3dbbd22c047c0d886532fa269ba"} Oct 11 10:56:28.063256 master-1 kubenswrapper[4771]: I1011 10:56:28.062890 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-api-0"] Oct 11 10:56:28.074635 master-1 kubenswrapper[4771]: I1011 10:56:28.074573 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell1-conductor-db-sync-tgg64"] Oct 11 10:56:28.076853 master-1 kubenswrapper[4771]: I1011 10:56:28.076835 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.080876 master-1 kubenswrapper[4771]: I1011 10:56:28.080824 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell1-conductor-scripts" Oct 11 10:56:28.080940 master-1 kubenswrapper[4771]: I1011 10:56:28.080845 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-conductor-db-sync-tgg64"] Oct 11 10:56:28.081153 master-1 kubenswrapper[4771]: I1011 10:56:28.081134 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell1-conductor-config-data" Oct 11 10:56:28.138127 master-1 kubenswrapper[4771]: I1011 10:56:28.136694 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5jhb\" (UniqueName: \"kubernetes.io/projected/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-kube-api-access-n5jhb\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.138127 master-1 kubenswrapper[4771]: I1011 10:56:28.136739 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-scripts\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.138127 master-1 kubenswrapper[4771]: I1011 10:56:28.136782 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-config-data\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.138127 master-1 kubenswrapper[4771]: I1011 10:56:28.136860 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-combined-ca-bundle\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.142035 master-1 kubenswrapper[4771]: I1011 10:56:28.141692 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell0-cell-mapping-2kt7k" podStartSLOduration=2.141663152 podStartE2EDuration="2.141663152s" podCreationTimestamp="2025-10-11 10:56:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:56:28.094329322 +0000 UTC m=+1820.068555763" watchObservedRunningTime="2025-10-11 10:56:28.141663152 +0000 UTC m=+1820.115889593" Oct 11 10:56:28.252956 master-1 kubenswrapper[4771]: I1011 10:56:28.225206 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:56:28.318176 master-1 kubenswrapper[4771]: I1011 10:56:28.318117 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-combined-ca-bundle\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.322535 master-1 kubenswrapper[4771]: I1011 10:56:28.322498 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-n5jhb\" (UniqueName: \"kubernetes.io/projected/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-kube-api-access-n5jhb\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.322620 master-1 kubenswrapper[4771]: I1011 10:56:28.322570 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-scripts\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.323232 master-1 kubenswrapper[4771]: I1011 10:56:28.323186 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-config-data\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.326999 master-1 kubenswrapper[4771]: I1011 10:56:28.326954 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-combined-ca-bundle\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.328348 master-1 kubenswrapper[4771]: I1011 10:56:28.328283 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-scripts\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.328837 master-1 kubenswrapper[4771]: I1011 10:56:28.328785 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-config-data\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.329538 master-1 kubenswrapper[4771]: I1011 10:56:28.329496 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-novncproxy-0"] Oct 11 10:56:28.355822 master-1 kubenswrapper[4771]: I1011 10:56:28.355720 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5jhb\" (UniqueName: \"kubernetes.io/projected/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-kube-api-access-n5jhb\") pod \"nova-cell1-conductor-db-sync-tgg64\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.360009 master-1 kubenswrapper[4771]: I1011 10:56:28.359947 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:56:28.510721 master-1 kubenswrapper[4771]: I1011 10:56:28.510612 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:28.976405 master-1 kubenswrapper[4771]: I1011 10:56:28.976316 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-conductor-db-sync-tgg64"] Oct 11 10:56:29.070503 master-1 kubenswrapper[4771]: I1011 10:56:29.070446 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"1596746b-25ca-487a-9e49-93e532f2838b","Type":"ContainerStarted","Data":"089dbb4c74bf7c97de17802c964262c34b58cf3f278035cc8b343f4df54f1f61"} Oct 11 10:56:29.072943 master-1 kubenswrapper[4771]: I1011 10:56:29.072674 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-0" event={"ID":"c648855c-73f8-4316-9eca-147821b776c2","Type":"ContainerStarted","Data":"901bf1754d9ccab6be8ecc339468f500f7f6d434ca5a6c15ca5caad9d817a352"} Oct 11 10:56:29.074728 master-1 kubenswrapper[4771]: I1011 10:56:29.074700 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-novncproxy-0" event={"ID":"602a8d3a-2ca2-43d2-8def-5718d9baf2ee","Type":"ContainerStarted","Data":"9b7f7d7c0af1b640b51f7a6b8a5687c5423669e8ea192915f2e34e079daaef17"} Oct 11 10:56:29.076544 master-1 kubenswrapper[4771]: I1011 10:56:29.076512 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-conductor-db-sync-tgg64" event={"ID":"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad","Type":"ContainerStarted","Data":"8225c71dadd5dd5d7cdb7b603f12129b97565dbfb98b6f8553a5f73b645e62cc"} Oct 11 10:56:29.077973 master-1 kubenswrapper[4771]: I1011 10:56:29.077939 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"4619bcb1-090e-4824-adfe-6a526158d0ea","Type":"ContainerStarted","Data":"2855d39e0600653f4ca98e1b1c4a631cd2cea811da489ab4ee595433738a99d4"} Oct 11 10:56:29.942384 master-1 kubenswrapper[4771]: I1011 10:56:29.940262 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ceilometer-0" Oct 11 10:56:30.112371 master-1 kubenswrapper[4771]: I1011 10:56:30.108710 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-conductor-db-sync-tgg64" event={"ID":"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad","Type":"ContainerStarted","Data":"af56a7e4623de207ef8289e7bba0d65eef5da9d57f459e288f321109c3a8e4f3"} Oct 11 10:56:30.151382 master-1 kubenswrapper[4771]: I1011 10:56:30.150237 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell1-conductor-db-sync-tgg64" podStartSLOduration=2.150148926 podStartE2EDuration="2.150148926s" podCreationTimestamp="2025-10-11 10:56:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:56:30.128858605 +0000 UTC m=+1822.103085046" watchObservedRunningTime="2025-10-11 10:56:30.150148926 +0000 UTC m=+1822.124375367" Oct 11 10:56:31.406381 master-1 kubenswrapper[4771]: I1011 10:56:31.405580 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-cell1-novncproxy-0"] Oct 11 10:56:33.585261 master-1 kubenswrapper[4771]: I1011 10:56:33.585179 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:33.585261 master-1 kubenswrapper[4771]: I1011 10:56:33.585265 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:33.618135 master-1 kubenswrapper[4771]: I1011 10:56:33.618077 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:33.643242 master-1 kubenswrapper[4771]: I1011 10:56:33.643195 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:34.154417 master-1 kubenswrapper[4771]: I1011 10:56:34.154294 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:34.154417 master-1 kubenswrapper[4771]: I1011 10:56:34.154403 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:35.175641 master-1 kubenswrapper[4771]: I1011 10:56:35.175322 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"4619bcb1-090e-4824-adfe-6a526158d0ea","Type":"ContainerStarted","Data":"a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8"} Oct 11 10:56:35.185376 master-1 kubenswrapper[4771]: I1011 10:56:35.184825 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"1596746b-25ca-487a-9e49-93e532f2838b","Type":"ContainerStarted","Data":"cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae"} Oct 11 10:56:35.185376 master-1 kubenswrapper[4771]: I1011 10:56:35.184900 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"1596746b-25ca-487a-9e49-93e532f2838b","Type":"ContainerStarted","Data":"10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f"} Oct 11 10:56:35.191381 master-1 kubenswrapper[4771]: I1011 10:56:35.191325 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-0" event={"ID":"c648855c-73f8-4316-9eca-147821b776c2","Type":"ContainerStarted","Data":"684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557"} Oct 11 10:56:35.191381 master-1 kubenswrapper[4771]: I1011 10:56:35.191367 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-0" event={"ID":"c648855c-73f8-4316-9eca-147821b776c2","Type":"ContainerStarted","Data":"88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6"} Oct 11 10:56:35.193500 master-1 kubenswrapper[4771]: I1011 10:56:35.193466 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-novncproxy-0" event={"ID":"602a8d3a-2ca2-43d2-8def-5718d9baf2ee","Type":"ContainerStarted","Data":"23ed8d2300382ac408577223ca4d96c6b722a1c94b3187394d9ec21991883547"} Oct 11 10:56:35.193652 master-1 kubenswrapper[4771]: I1011 10:56:35.193586 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/nova-cell1-novncproxy-0" podUID="602a8d3a-2ca2-43d2-8def-5718d9baf2ee" containerName="nova-cell1-novncproxy-novncproxy" containerID="cri-o://23ed8d2300382ac408577223ca4d96c6b722a1c94b3187394d9ec21991883547" gracePeriod=30 Oct 11 10:56:35.199571 master-1 kubenswrapper[4771]: I1011 10:56:35.199485 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-db-sync-tn8xz" event={"ID":"3de492fb-5249-49e2-a327-756234aa92bd","Type":"ContainerStarted","Data":"38fe7e740cc7430b1900679565564cc35f6e1964bf7c4a238c960c0377445331"} Oct 11 10:56:35.211692 master-1 kubenswrapper[4771]: I1011 10:56:35.211599 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-scheduler-1" podStartSLOduration=2.431765014 podStartE2EDuration="8.21157798s" podCreationTimestamp="2025-10-11 10:56:27 +0000 UTC" firstStartedPulling="2025-10-11 10:56:28.313837973 +0000 UTC m=+1820.288064414" lastFinishedPulling="2025-10-11 10:56:34.093650939 +0000 UTC m=+1826.067877380" observedRunningTime="2025-10-11 10:56:35.206577324 +0000 UTC m=+1827.180803785" watchObservedRunningTime="2025-10-11 10:56:35.21157798 +0000 UTC m=+1827.185804421" Oct 11 10:56:36.085447 master-1 kubenswrapper[4771]: I1011 10:56:36.084809 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-metadata-1" podStartSLOduration=3.359572071 podStartE2EDuration="9.084772235s" podCreationTimestamp="2025-10-11 10:56:27 +0000 UTC" firstStartedPulling="2025-10-11 10:56:28.368023493 +0000 UTC m=+1820.342249934" lastFinishedPulling="2025-10-11 10:56:34.093223647 +0000 UTC m=+1826.067450098" observedRunningTime="2025-10-11 10:56:36.038694472 +0000 UTC m=+1828.012920993" watchObservedRunningTime="2025-10-11 10:56:36.084772235 +0000 UTC m=+1828.058998676" Oct 11 10:56:36.090912 master-1 kubenswrapper[4771]: I1011 10:56:36.088220 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell1-novncproxy-0" podStartSLOduration=3.313161298 podStartE2EDuration="9.088205085s" podCreationTimestamp="2025-10-11 10:56:27 +0000 UTC" firstStartedPulling="2025-10-11 10:56:28.327914044 +0000 UTC m=+1820.302140485" lastFinishedPulling="2025-10-11 10:56:34.102957821 +0000 UTC m=+1826.077184272" observedRunningTime="2025-10-11 10:56:36.077536474 +0000 UTC m=+1828.051762925" watchObservedRunningTime="2025-10-11 10:56:36.088205085 +0000 UTC m=+1828.062431526" Oct 11 10:56:36.113564 master-1 kubenswrapper[4771]: I1011 10:56:36.113321 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:36.130101 master-1 kubenswrapper[4771]: I1011 10:56:36.129987 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/aodh-db-sync-tn8xz" podStartSLOduration=2.786464607 podStartE2EDuration="10.129955633s" podCreationTimestamp="2025-10-11 10:56:26 +0000 UTC" firstStartedPulling="2025-10-11 10:56:26.907478891 +0000 UTC m=+1818.881705342" lastFinishedPulling="2025-10-11 10:56:34.250969927 +0000 UTC m=+1826.225196368" observedRunningTime="2025-10-11 10:56:36.110264029 +0000 UTC m=+1828.084490490" watchObservedRunningTime="2025-10-11 10:56:36.129955633 +0000 UTC m=+1828.104182084" Oct 11 10:56:36.145751 master-1 kubenswrapper[4771]: I1011 10:56:36.145634 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-api-0" podStartSLOduration=3.106001827 podStartE2EDuration="9.145605939s" podCreationTimestamp="2025-10-11 10:56:27 +0000 UTC" firstStartedPulling="2025-10-11 10:56:28.057290692 +0000 UTC m=+1820.031517133" lastFinishedPulling="2025-10-11 10:56:34.096894774 +0000 UTC m=+1826.071121245" observedRunningTime="2025-10-11 10:56:36.138105981 +0000 UTC m=+1828.112332422" watchObservedRunningTime="2025-10-11 10:56:36.145605939 +0000 UTC m=+1828.119832390" Oct 11 10:56:36.210612 master-1 kubenswrapper[4771]: I1011 10:56:36.210531 4771 generic.go:334] "Generic (PLEG): container finished" podID="f85d5cfa-8073-4bbf-9eff-78fde719dadf" containerID="03e80f5bdb6844a3112427ed3612b145765c86f689c582771359401e14c9758e" exitCode=0 Oct 11 10:56:36.211878 master-1 kubenswrapper[4771]: I1011 10:56:36.211005 4771 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Oct 11 10:56:36.211878 master-1 kubenswrapper[4771]: I1011 10:56:36.211551 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-cell-mapping-2kt7k" event={"ID":"f85d5cfa-8073-4bbf-9eff-78fde719dadf","Type":"ContainerDied","Data":"03e80f5bdb6844a3112427ed3612b145765c86f689c582771359401e14c9758e"} Oct 11 10:56:36.362086 master-1 kubenswrapper[4771]: I1011 10:56:36.359579 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:36.362086 master-1 kubenswrapper[4771]: I1011 10:56:36.360318 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="sg-core" containerID="cri-o://dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68" gracePeriod=30 Oct 11 10:56:36.362086 master-1 kubenswrapper[4771]: I1011 10:56:36.360322 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="proxy-httpd" containerID="cri-o://505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe" gracePeriod=30 Oct 11 10:56:36.362086 master-1 kubenswrapper[4771]: I1011 10:56:36.360368 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="ceilometer-notification-agent" containerID="cri-o://6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111" gracePeriod=30 Oct 11 10:56:36.362086 master-1 kubenswrapper[4771]: I1011 10:56:36.360633 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="ceilometer-central-agent" containerID="cri-o://2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536" gracePeriod=30 Oct 11 10:56:36.456922 master-1 kubenswrapper[4771]: I1011 10:56:36.456509 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/glance-b5802-default-external-api-0" Oct 11 10:56:37.235768 master-1 kubenswrapper[4771]: I1011 10:56:37.234806 4771 generic.go:334] "Generic (PLEG): container finished" podID="3de492fb-5249-49e2-a327-756234aa92bd" containerID="38fe7e740cc7430b1900679565564cc35f6e1964bf7c4a238c960c0377445331" exitCode=0 Oct 11 10:56:37.236501 master-1 kubenswrapper[4771]: I1011 10:56:37.235756 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-db-sync-tn8xz" event={"ID":"3de492fb-5249-49e2-a327-756234aa92bd","Type":"ContainerDied","Data":"38fe7e740cc7430b1900679565564cc35f6e1964bf7c4a238c960c0377445331"} Oct 11 10:56:37.241674 master-1 kubenswrapper[4771]: I1011 10:56:37.241455 4771 generic.go:334] "Generic (PLEG): container finished" podID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerID="505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe" exitCode=0 Oct 11 10:56:37.241674 master-1 kubenswrapper[4771]: I1011 10:56:37.241568 4771 generic.go:334] "Generic (PLEG): container finished" podID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerID="dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68" exitCode=2 Oct 11 10:56:37.241674 master-1 kubenswrapper[4771]: I1011 10:56:37.241580 4771 generic.go:334] "Generic (PLEG): container finished" podID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerID="2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536" exitCode=0 Oct 11 10:56:37.243665 master-1 kubenswrapper[4771]: I1011 10:56:37.243529 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerDied","Data":"505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe"} Oct 11 10:56:37.243951 master-1 kubenswrapper[4771]: I1011 10:56:37.243924 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerDied","Data":"dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68"} Oct 11 10:56:37.244016 master-1 kubenswrapper[4771]: I1011 10:56:37.243953 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerDied","Data":"2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536"} Oct 11 10:56:37.390607 master-1 kubenswrapper[4771]: W1011 10:56:37.390534 4771 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb670525b_9ca9_419c_858b_6bb2a2303cf6.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb670525b_9ca9_419c_858b_6bb2a2303cf6.slice: no such file or directory Oct 11 10:56:37.517160 master-1 kubenswrapper[4771]: E1011 10:56:37.517052 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-48979e1f4a2c3ea1ab7c6b56d19ad482a69e0c44021b08cc343a04080547c276\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-conmon-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-conmon-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice\": RecentStats: unable to find data in memory cache]" Oct 11 10:56:37.525280 master-1 kubenswrapper[4771]: E1011 10:56:37.525177 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-a58cd7c5986caf2dc23489be67139cc61da2a0165758a7758f40955bc3fa5183\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-conmon-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-conmon-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-48979e1f4a2c3ea1ab7c6b56d19ad482a69e0c44021b08cc343a04080547c276\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-conmon-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:56:37.526015 master-1 kubenswrapper[4771]: E1011 10:56:37.525561 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-conmon-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-conmon-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-conmon-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-a58cd7c5986caf2dc23489be67139cc61da2a0165758a7758f40955bc3fa5183\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-48979e1f4a2c3ea1ab7c6b56d19ad482a69e0c44021b08cc343a04080547c276\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:56:37.535019 master-1 kubenswrapper[4771]: E1011 10:56:37.534913 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-conmon-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-conmon-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-48979e1f4a2c3ea1ab7c6b56d19ad482a69e0c44021b08cc343a04080547c276\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-a58cd7c5986caf2dc23489be67139cc61da2a0165758a7758f40955bc3fa5183\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-conmon-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:56:37.538393 master-1 kubenswrapper[4771]: E1011 10:56:37.538269 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-conmon-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-48979e1f4a2c3ea1ab7c6b56d19ad482a69e0c44021b08cc343a04080547c276\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-conmon-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-a58cd7c5986caf2dc23489be67139cc61da2a0165758a7758f40955bc3fa5183\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-conmon-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe\": RecentStats: unable to find data in memory cache]" Oct 11 10:56:37.540713 master-1 kubenswrapper[4771]: E1011 10:56:37.539164 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-conmon-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-conmon-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-a58cd7c5986caf2dc23489be67139cc61da2a0165758a7758f40955bc3fa5183\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-48979e1f4a2c3ea1ab7c6b56d19ad482a69e0c44021b08cc343a04080547c276\": RecentStats: unable to find data in memory cache]" Oct 11 10:56:37.542538 master-1 kubenswrapper[4771]: E1011 10:56:37.542458 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-conmon-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-conmon-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-a58cd7c5986caf2dc23489be67139cc61da2a0165758a7758f40955bc3fa5183\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe\": RecentStats: unable to find data in memory cache]" Oct 11 10:56:37.545320 master-1 kubenswrapper[4771]: E1011 10:56:37.545177 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-conmon-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice/crio-0c0dbb46c9c9dd56fc370dbf3bf4cefb3705db5d800f0bb6bbcaf05c4ff29fa5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-conmon-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-a58cd7c5986caf2dc23489be67139cc61da2a0165758a7758f40955bc3fa5183\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-fa2fab9dc9ec47645ed24513cf6556f25abd1fec54df11a83d3b7a83c2bf8f53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-conmon-7a81fdca6b5cfdc52d40a0c549f548b156cef8be2d800941e09638f91911a474.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3028266_255a_43a3_8bdb_9695ad7cbb30.slice/crio-9147911025dd8a83203254747854ba1cbd5aacf26ee1c4c2cc04460690d1b34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-48979e1f4a2c3ea1ab7c6b56d19ad482a69e0c44021b08cc343a04080547c276\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921e565_c581_42f4_8da8_df72fae9a3c0.slice/crio-2046d1999ef071c637c21e7f0dabc3a11726315f639116dfe75b51029d76c1fe\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38267a66_0ebd_44ab_bc7f_cd5703503b74.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478147ef_a0d7_4c37_952c_3fc3a23775db.slice/crio-cbc52f8e44c15d5b309b9a5f8920e77aecfb6aec423daa8801871f7c83313c80.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:56:37.553403 master-1 kubenswrapper[4771]: I1011 10:56:37.552556 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-api-0" Oct 11 10:56:37.553403 master-1 kubenswrapper[4771]: I1011 10:56:37.552619 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-api-0" Oct 11 10:56:37.582439 master-1 kubenswrapper[4771]: I1011 10:56:37.580583 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-scheduler-1" Oct 11 10:56:37.582439 master-1 kubenswrapper[4771]: I1011 10:56:37.582039 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-scheduler-1" Oct 11 10:56:37.622286 master-1 kubenswrapper[4771]: I1011 10:56:37.622241 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-scheduler-1" Oct 11 10:56:37.780822 master-1 kubenswrapper[4771]: I1011 10:56:37.780756 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:56:37.969134 master-1 kubenswrapper[4771]: I1011 10:56:37.968988 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-metadata-1" Oct 11 10:56:37.969968 master-1 kubenswrapper[4771]: I1011 10:56:37.969938 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-metadata-1" Oct 11 10:56:37.970059 master-1 kubenswrapper[4771]: I1011 10:56:37.969978 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-metadata-1" Oct 11 10:56:37.970059 master-1 kubenswrapper[4771]: I1011 10:56:37.969991 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-metadata-1" Oct 11 10:56:38.270383 master-1 kubenswrapper[4771]: I1011 10:56:38.266870 4771 generic.go:334] "Generic (PLEG): container finished" podID="1fe7833d-9251-4545-ba68-f58c146188f1" containerID="3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d" exitCode=137 Oct 11 10:56:38.270383 master-1 kubenswrapper[4771]: I1011 10:56:38.267193 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" event={"ID":"1fe7833d-9251-4545-ba68-f58c146188f1","Type":"ContainerDied","Data":"3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d"} Oct 11 10:56:38.287380 master-1 kubenswrapper[4771]: I1011 10:56:38.286005 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-79cbf74f6f-j7kt4"] Oct 11 10:56:38.293378 master-1 kubenswrapper[4771]: I1011 10:56:38.288636 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.311372 master-1 kubenswrapper[4771]: I1011 10:56:38.307958 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-79cbf74f6f-j7kt4"] Oct 11 10:56:38.365182 master-1 kubenswrapper[4771]: I1011 10:56:38.352582 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-svc\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.365182 master-1 kubenswrapper[4771]: I1011 10:56:38.352678 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-nb\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.365182 master-1 kubenswrapper[4771]: I1011 10:56:38.352728 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-sb\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.365182 master-1 kubenswrapper[4771]: I1011 10:56:38.352746 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-swift-storage-0\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.365182 master-1 kubenswrapper[4771]: I1011 10:56:38.352777 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7n6h\" (UniqueName: \"kubernetes.io/projected/a23d84be-f5ab-4261-9ba2-d94aaf104a59-kube-api-access-k7n6h\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.365182 master-1 kubenswrapper[4771]: I1011 10:56:38.352852 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-config\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.380374 master-1 kubenswrapper[4771]: I1011 10:56:38.375931 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-scheduler-1" Oct 11 10:56:38.460380 master-1 kubenswrapper[4771]: I1011 10:56:38.458212 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-nb\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.460380 master-1 kubenswrapper[4771]: I1011 10:56:38.458296 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-sb\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.460380 master-1 kubenswrapper[4771]: I1011 10:56:38.458334 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-swift-storage-0\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.460380 master-1 kubenswrapper[4771]: I1011 10:56:38.458391 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-k7n6h\" (UniqueName: \"kubernetes.io/projected/a23d84be-f5ab-4261-9ba2-d94aaf104a59-kube-api-access-k7n6h\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.460380 master-1 kubenswrapper[4771]: I1011 10:56:38.458490 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-config\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.460380 master-1 kubenswrapper[4771]: I1011 10:56:38.458573 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-svc\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.460380 master-1 kubenswrapper[4771]: I1011 10:56:38.459418 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-svc\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.460380 master-1 kubenswrapper[4771]: I1011 10:56:38.459982 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-nb\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.464368 master-1 kubenswrapper[4771]: I1011 10:56:38.460915 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-sb\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.464368 master-1 kubenswrapper[4771]: I1011 10:56:38.461882 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-config\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.464368 master-1 kubenswrapper[4771]: I1011 10:56:38.461995 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-swift-storage-0\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.503376 master-1 kubenswrapper[4771]: I1011 10:56:38.503218 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7n6h\" (UniqueName: \"kubernetes.io/projected/a23d84be-f5ab-4261-9ba2-d94aaf104a59-kube-api-access-k7n6h\") pod \"dnsmasq-dns-79cbf74f6f-j7kt4\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.616388 master-1 kubenswrapper[4771]: I1011 10:56:38.612623 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:38.636377 master-1 kubenswrapper[4771]: I1011 10:56:38.634555 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-api-0" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-api" probeResult="failure" output="Get \"http://10.129.0.157:8774/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 11 10:56:38.636377 master-1 kubenswrapper[4771]: I1011 10:56:38.634563 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-api-0" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-log" probeResult="failure" output="Get \"http://10.129.0.157:8774/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 11 10:56:39.010388 master-1 kubenswrapper[4771]: I1011 10:56:39.009692 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-metadata-1" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-log" probeResult="failure" output="Get \"http://10.129.0.160:8775/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 11 10:56:39.051138 master-1 kubenswrapper[4771]: I1011 10:56:39.050561 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-metadata-1" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-metadata" probeResult="failure" output="Get \"http://10.129.0.160:8775/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 11 10:56:39.282455 master-1 kubenswrapper[4771]: I1011 10:56:39.281707 4771 generic.go:334] "Generic (PLEG): container finished" podID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerID="6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111" exitCode=0 Oct 11 10:56:39.282455 master-1 kubenswrapper[4771]: I1011 10:56:39.281848 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerDied","Data":"6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111"} Oct 11 10:56:41.302278 master-1 kubenswrapper[4771]: I1011 10:56:41.302198 4771 generic.go:334] "Generic (PLEG): container finished" podID="5b5e37e3-9afd-4ff3-b992-1e6c28a986ad" containerID="af56a7e4623de207ef8289e7bba0d65eef5da9d57f459e288f321109c3a8e4f3" exitCode=0 Oct 11 10:56:41.302278 master-1 kubenswrapper[4771]: I1011 10:56:41.302244 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-conductor-db-sync-tgg64" event={"ID":"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad","Type":"ContainerDied","Data":"af56a7e4623de207ef8289e7bba0d65eef5da9d57f459e288f321109c3a8e4f3"} Oct 11 10:56:41.374117 master-1 kubenswrapper[4771]: I1011 10:56:41.374031 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" podUID="1fe7833d-9251-4545-ba68-f58c146188f1" containerName="heat-cfnapi" probeResult="failure" output="Get \"http://10.129.0.137:8000/healthcheck\": dial tcp 10.129.0.137:8000: connect: connection refused" Oct 11 10:56:41.787338 master-1 kubenswrapper[4771]: I1011 10:56:41.787245 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:41.833428 master-1 kubenswrapper[4771]: I1011 10:56:41.831959 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:41.884015 master-1 kubenswrapper[4771]: I1011 10:56:41.883537 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mgq2\" (UniqueName: \"kubernetes.io/projected/3de492fb-5249-49e2-a327-756234aa92bd-kube-api-access-6mgq2\") pod \"3de492fb-5249-49e2-a327-756234aa92bd\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " Oct 11 10:56:41.884015 master-1 kubenswrapper[4771]: I1011 10:56:41.883740 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-config-data\") pod \"3de492fb-5249-49e2-a327-756234aa92bd\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " Oct 11 10:56:41.884015 master-1 kubenswrapper[4771]: I1011 10:56:41.883831 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-combined-ca-bundle\") pod \"3de492fb-5249-49e2-a327-756234aa92bd\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " Oct 11 10:56:41.884015 master-1 kubenswrapper[4771]: I1011 10:56:41.883895 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-scripts\") pod \"3de492fb-5249-49e2-a327-756234aa92bd\" (UID: \"3de492fb-5249-49e2-a327-756234aa92bd\") " Oct 11 10:56:41.888739 master-1 kubenswrapper[4771]: I1011 10:56:41.888656 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-scripts" (OuterVolumeSpecName: "scripts") pod "3de492fb-5249-49e2-a327-756234aa92bd" (UID: "3de492fb-5249-49e2-a327-756234aa92bd"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:41.890460 master-1 kubenswrapper[4771]: I1011 10:56:41.889440 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de492fb-5249-49e2-a327-756234aa92bd-kube-api-access-6mgq2" (OuterVolumeSpecName: "kube-api-access-6mgq2") pod "3de492fb-5249-49e2-a327-756234aa92bd" (UID: "3de492fb-5249-49e2-a327-756234aa92bd"). InnerVolumeSpecName "kube-api-access-6mgq2". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:41.920683 master-1 kubenswrapper[4771]: I1011 10:56:41.920626 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-config-data" (OuterVolumeSpecName: "config-data") pod "3de492fb-5249-49e2-a327-756234aa92bd" (UID: "3de492fb-5249-49e2-a327-756234aa92bd"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:41.936736 master-1 kubenswrapper[4771]: I1011 10:56:41.936674 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "3de492fb-5249-49e2-a327-756234aa92bd" (UID: "3de492fb-5249-49e2-a327-756234aa92bd"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:41.988080 master-1 kubenswrapper[4771]: I1011 10:56:41.987755 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clg6t\" (UniqueName: \"kubernetes.io/projected/f85d5cfa-8073-4bbf-9eff-78fde719dadf-kube-api-access-clg6t\") pod \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " Oct 11 10:56:41.988080 master-1 kubenswrapper[4771]: I1011 10:56:41.987922 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-scripts\") pod \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " Oct 11 10:56:41.988080 master-1 kubenswrapper[4771]: I1011 10:56:41.987942 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-config-data\") pod \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " Oct 11 10:56:41.988080 master-1 kubenswrapper[4771]: I1011 10:56:41.988028 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-combined-ca-bundle\") pod \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\" (UID: \"f85d5cfa-8073-4bbf-9eff-78fde719dadf\") " Oct 11 10:56:41.989185 master-1 kubenswrapper[4771]: I1011 10:56:41.988444 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:41.989185 master-1 kubenswrapper[4771]: I1011 10:56:41.988461 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-6mgq2\" (UniqueName: \"kubernetes.io/projected/3de492fb-5249-49e2-a327-756234aa92bd-kube-api-access-6mgq2\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:41.989185 master-1 kubenswrapper[4771]: I1011 10:56:41.988471 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:41.989185 master-1 kubenswrapper[4771]: I1011 10:56:41.988479 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3de492fb-5249-49e2-a327-756234aa92bd-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:41.992843 master-1 kubenswrapper[4771]: I1011 10:56:41.992785 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-scripts" (OuterVolumeSpecName: "scripts") pod "f85d5cfa-8073-4bbf-9eff-78fde719dadf" (UID: "f85d5cfa-8073-4bbf-9eff-78fde719dadf"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:41.996234 master-1 kubenswrapper[4771]: I1011 10:56:41.996156 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85d5cfa-8073-4bbf-9eff-78fde719dadf-kube-api-access-clg6t" (OuterVolumeSpecName: "kube-api-access-clg6t") pod "f85d5cfa-8073-4bbf-9eff-78fde719dadf" (UID: "f85d5cfa-8073-4bbf-9eff-78fde719dadf"). InnerVolumeSpecName "kube-api-access-clg6t". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:42.038245 master-1 kubenswrapper[4771]: I1011 10:56:42.038172 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "f85d5cfa-8073-4bbf-9eff-78fde719dadf" (UID: "f85d5cfa-8073-4bbf-9eff-78fde719dadf"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.040560 master-1 kubenswrapper[4771]: I1011 10:56:42.040493 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-config-data" (OuterVolumeSpecName: "config-data") pod "f85d5cfa-8073-4bbf-9eff-78fde719dadf" (UID: "f85d5cfa-8073-4bbf-9eff-78fde719dadf"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.066724 master-1 kubenswrapper[4771]: I1011 10:56:42.064847 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:56:42.090958 master-1 kubenswrapper[4771]: I1011 10:56:42.090906 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.091220 master-1 kubenswrapper[4771]: I1011 10:56:42.091207 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.091317 master-1 kubenswrapper[4771]: I1011 10:56:42.091302 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f85d5cfa-8073-4bbf-9eff-78fde719dadf-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.091474 master-1 kubenswrapper[4771]: I1011 10:56:42.091459 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-clg6t\" (UniqueName: \"kubernetes.io/projected/f85d5cfa-8073-4bbf-9eff-78fde719dadf-kube-api-access-clg6t\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.172275 master-1 kubenswrapper[4771]: I1011 10:56:42.172203 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:56:42.193141 master-1 kubenswrapper[4771]: I1011 10:56:42.193062 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-combined-ca-bundle\") pod \"1fe7833d-9251-4545-ba68-f58c146188f1\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " Oct 11 10:56:42.194031 master-1 kubenswrapper[4771]: I1011 10:56:42.193263 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzbk4\" (UniqueName: \"kubernetes.io/projected/1fe7833d-9251-4545-ba68-f58c146188f1-kube-api-access-gzbk4\") pod \"1fe7833d-9251-4545-ba68-f58c146188f1\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " Oct 11 10:56:42.194031 master-1 kubenswrapper[4771]: I1011 10:56:42.193334 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data-custom\") pod \"1fe7833d-9251-4545-ba68-f58c146188f1\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " Oct 11 10:56:42.194031 master-1 kubenswrapper[4771]: I1011 10:56:42.193445 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data\") pod \"1fe7833d-9251-4545-ba68-f58c146188f1\" (UID: \"1fe7833d-9251-4545-ba68-f58c146188f1\") " Oct 11 10:56:42.197467 master-1 kubenswrapper[4771]: I1011 10:56:42.197424 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data-custom" (OuterVolumeSpecName: "config-data-custom") pod "1fe7833d-9251-4545-ba68-f58c146188f1" (UID: "1fe7833d-9251-4545-ba68-f58c146188f1"). InnerVolumeSpecName "config-data-custom". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.197998 master-1 kubenswrapper[4771]: I1011 10:56:42.197923 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe7833d-9251-4545-ba68-f58c146188f1-kube-api-access-gzbk4" (OuterVolumeSpecName: "kube-api-access-gzbk4") pod "1fe7833d-9251-4545-ba68-f58c146188f1" (UID: "1fe7833d-9251-4545-ba68-f58c146188f1"). InnerVolumeSpecName "kube-api-access-gzbk4". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:42.219384 master-1 kubenswrapper[4771]: I1011 10:56:42.219313 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "1fe7833d-9251-4545-ba68-f58c146188f1" (UID: "1fe7833d-9251-4545-ba68-f58c146188f1"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.233412 master-1 kubenswrapper[4771]: I1011 10:56:42.233342 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data" (OuterVolumeSpecName: "config-data") pod "1fe7833d-9251-4545-ba68-f58c146188f1" (UID: "1fe7833d-9251-4545-ba68-f58c146188f1"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.294781 master-1 kubenswrapper[4771]: I1011 10:56:42.294735 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75dsl\" (UniqueName: \"kubernetes.io/projected/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-kube-api-access-75dsl\") pod \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " Oct 11 10:56:42.294930 master-1 kubenswrapper[4771]: I1011 10:56:42.294884 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-sg-core-conf-yaml\") pod \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " Oct 11 10:56:42.294979 master-1 kubenswrapper[4771]: I1011 10:56:42.294948 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-config-data\") pod \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " Oct 11 10:56:42.295058 master-1 kubenswrapper[4771]: I1011 10:56:42.295002 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-combined-ca-bundle\") pod \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " Oct 11 10:56:42.295102 master-1 kubenswrapper[4771]: I1011 10:56:42.295073 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-run-httpd\") pod \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " Oct 11 10:56:42.295102 master-1 kubenswrapper[4771]: I1011 10:56:42.295097 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-scripts\") pod \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " Oct 11 10:56:42.295190 master-1 kubenswrapper[4771]: I1011 10:56:42.295119 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-log-httpd\") pod \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\" (UID: \"d0cc5394-b33f-41a9-bbe2-d772e75a8f58\") " Oct 11 10:56:42.295525 master-1 kubenswrapper[4771]: I1011 10:56:42.295489 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.295525 master-1 kubenswrapper[4771]: I1011 10:56:42.295507 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.295525 master-1 kubenswrapper[4771]: I1011 10:56:42.295517 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-gzbk4\" (UniqueName: \"kubernetes.io/projected/1fe7833d-9251-4545-ba68-f58c146188f1-kube-api-access-gzbk4\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.295525 master-1 kubenswrapper[4771]: I1011 10:56:42.295529 4771 reconciler_common.go:293] "Volume detached for volume \"config-data-custom\" (UniqueName: \"kubernetes.io/secret/1fe7833d-9251-4545-ba68-f58c146188f1-config-data-custom\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.295888 master-1 kubenswrapper[4771]: I1011 10:56:42.295853 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "d0cc5394-b33f-41a9-bbe2-d772e75a8f58" (UID: "d0cc5394-b33f-41a9-bbe2-d772e75a8f58"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:42.296137 master-1 kubenswrapper[4771]: I1011 10:56:42.296101 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "d0cc5394-b33f-41a9-bbe2-d772e75a8f58" (UID: "d0cc5394-b33f-41a9-bbe2-d772e75a8f58"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:42.298532 master-1 kubenswrapper[4771]: I1011 10:56:42.298493 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-kube-api-access-75dsl" (OuterVolumeSpecName: "kube-api-access-75dsl") pod "d0cc5394-b33f-41a9-bbe2-d772e75a8f58" (UID: "d0cc5394-b33f-41a9-bbe2-d772e75a8f58"). InnerVolumeSpecName "kube-api-access-75dsl". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:42.298676 master-1 kubenswrapper[4771]: I1011 10:56:42.298649 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-scripts" (OuterVolumeSpecName: "scripts") pod "d0cc5394-b33f-41a9-bbe2-d772e75a8f58" (UID: "d0cc5394-b33f-41a9-bbe2-d772e75a8f58"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.314465 master-1 kubenswrapper[4771]: I1011 10:56:42.314348 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" event={"ID":"1fe7833d-9251-4545-ba68-f58c146188f1","Type":"ContainerDied","Data":"fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e"} Oct 11 10:56:42.314926 master-1 kubenswrapper[4771]: I1011 10:56:42.314477 4771 scope.go:117] "RemoveContainer" containerID="3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d" Oct 11 10:56:42.314926 master-1 kubenswrapper[4771]: I1011 10:56:42.314675 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/heat-cfnapi-b8cb664c5-5zrqf" Oct 11 10:56:42.318711 master-1 kubenswrapper[4771]: I1011 10:56:42.318660 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d0cc5394-b33f-41a9-bbe2-d772e75a8f58","Type":"ContainerDied","Data":"c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0"} Oct 11 10:56:42.318816 master-1 kubenswrapper[4771]: I1011 10:56:42.318794 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:56:42.321259 master-1 kubenswrapper[4771]: I1011 10:56:42.321218 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-compute-ironic-compute-0" event={"ID":"3a8c3af6-0b6a-486d-83e3-18bf00346dbc","Type":"ContainerStarted","Data":"618dbb2c6015f37d5b77c36226ff32674b1cdb27c9a2c24d18db48b89442be0b"} Oct 11 10:56:42.322221 master-1 kubenswrapper[4771]: I1011 10:56:42.322198 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:42.323429 master-1 kubenswrapper[4771]: I1011 10:56:42.323405 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/aodh-db-sync-tn8xz" event={"ID":"3de492fb-5249-49e2-a327-756234aa92bd","Type":"ContainerDied","Data":"4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344"} Oct 11 10:56:42.323429 master-1 kubenswrapper[4771]: I1011 10:56:42.323430 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344" Oct 11 10:56:42.323570 master-1 kubenswrapper[4771]: I1011 10:56:42.323472 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/aodh-db-sync-tn8xz" Oct 11 10:56:42.338345 master-1 kubenswrapper[4771]: I1011 10:56:42.338232 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell0-cell-mapping-2kt7k" event={"ID":"f85d5cfa-8073-4bbf-9eff-78fde719dadf","Type":"ContainerDied","Data":"792c6e93eebb0025120939bb299c7a87876ec3dbbd22c047c0d886532fa269ba"} Oct 11 10:56:42.338572 master-1 kubenswrapper[4771]: I1011 10:56:42.338412 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="792c6e93eebb0025120939bb299c7a87876ec3dbbd22c047c0d886532fa269ba" Oct 11 10:56:42.338572 master-1 kubenswrapper[4771]: I1011 10:56:42.338301 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell0-cell-mapping-2kt7k" Oct 11 10:56:42.339272 master-1 kubenswrapper[4771]: I1011 10:56:42.339215 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "d0cc5394-b33f-41a9-bbe2-d772e75a8f58" (UID: "d0cc5394-b33f-41a9-bbe2-d772e75a8f58"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.339775 master-1 kubenswrapper[4771]: I1011 10:56:42.339721 4771 scope.go:117] "RemoveContainer" containerID="505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe" Oct 11 10:56:42.354884 master-1 kubenswrapper[4771]: I1011 10:56:42.354834 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-cell1-compute-ironic-compute-0" Oct 11 10:56:42.371502 master-1 kubenswrapper[4771]: I1011 10:56:42.371456 4771 scope.go:117] "RemoveContainer" containerID="dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68" Oct 11 10:56:42.379281 master-1 kubenswrapper[4771]: I1011 10:56:42.379222 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "d0cc5394-b33f-41a9-bbe2-d772e75a8f58" (UID: "d0cc5394-b33f-41a9-bbe2-d772e75a8f58"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.397743 master-1 kubenswrapper[4771]: I1011 10:56:42.397698 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.397743 master-1 kubenswrapper[4771]: I1011 10:56:42.397743 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.397901 master-1 kubenswrapper[4771]: I1011 10:56:42.397753 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.397901 master-1 kubenswrapper[4771]: I1011 10:56:42.397763 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.397901 master-1 kubenswrapper[4771]: I1011 10:56:42.397772 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-75dsl\" (UniqueName: \"kubernetes.io/projected/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-kube-api-access-75dsl\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.397901 master-1 kubenswrapper[4771]: I1011 10:56:42.397783 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.399745 master-1 kubenswrapper[4771]: I1011 10:56:42.399687 4771 scope.go:117] "RemoveContainer" containerID="6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111" Oct 11 10:56:42.402038 master-1 kubenswrapper[4771]: I1011 10:56:42.401936 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-config-data" (OuterVolumeSpecName: "config-data") pod "d0cc5394-b33f-41a9-bbe2-d772e75a8f58" (UID: "d0cc5394-b33f-41a9-bbe2-d772e75a8f58"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.421622 master-1 kubenswrapper[4771]: I1011 10:56:42.421570 4771 scope.go:117] "RemoveContainer" containerID="2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536" Oct 11 10:56:42.500589 master-1 kubenswrapper[4771]: I1011 10:56:42.500524 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d0cc5394-b33f-41a9-bbe2-d772e75a8f58-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:42.833519 master-1 kubenswrapper[4771]: I1011 10:56:42.833456 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:42.910099 master-1 kubenswrapper[4771]: I1011 10:56:42.908440 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-config-data\") pod \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " Oct 11 10:56:42.910099 master-1 kubenswrapper[4771]: I1011 10:56:42.908552 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5jhb\" (UniqueName: \"kubernetes.io/projected/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-kube-api-access-n5jhb\") pod \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " Oct 11 10:56:42.910099 master-1 kubenswrapper[4771]: I1011 10:56:42.908658 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-combined-ca-bundle\") pod \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " Oct 11 10:56:42.910099 master-1 kubenswrapper[4771]: I1011 10:56:42.908864 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-scripts\") pod \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\" (UID: \"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad\") " Oct 11 10:56:42.914018 master-1 kubenswrapper[4771]: I1011 10:56:42.913564 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-scripts" (OuterVolumeSpecName: "scripts") pod "5b5e37e3-9afd-4ff3-b992-1e6c28a986ad" (UID: "5b5e37e3-9afd-4ff3-b992-1e6c28a986ad"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.914504 master-1 kubenswrapper[4771]: I1011 10:56:42.914388 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-79cbf74f6f-j7kt4"] Oct 11 10:56:42.918743 master-1 kubenswrapper[4771]: I1011 10:56:42.918705 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-kube-api-access-n5jhb" (OuterVolumeSpecName: "kube-api-access-n5jhb") pod "5b5e37e3-9afd-4ff3-b992-1e6c28a986ad" (UID: "5b5e37e3-9afd-4ff3-b992-1e6c28a986ad"). InnerVolumeSpecName "kube-api-access-n5jhb". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:42.938420 master-1 kubenswrapper[4771]: I1011 10:56:42.938287 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-config-data" (OuterVolumeSpecName: "config-data") pod "5b5e37e3-9afd-4ff3-b992-1e6c28a986ad" (UID: "5b5e37e3-9afd-4ff3-b992-1e6c28a986ad"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:42.944640 master-1 kubenswrapper[4771]: I1011 10:56:42.944558 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "5b5e37e3-9afd-4ff3-b992-1e6c28a986ad" (UID: "5b5e37e3-9afd-4ff3-b992-1e6c28a986ad"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:43.010993 master-1 kubenswrapper[4771]: I1011 10:56:43.010912 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:43.010993 master-1 kubenswrapper[4771]: I1011 10:56:43.010975 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:43.010993 master-1 kubenswrapper[4771]: I1011 10:56:43.010991 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-n5jhb\" (UniqueName: \"kubernetes.io/projected/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-kube-api-access-n5jhb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:43.010993 master-1 kubenswrapper[4771]: I1011 10:56:43.011004 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:43.198868 master-1 kubenswrapper[4771]: I1011 10:56:43.198639 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell1-compute-ironic-compute-0" podStartSLOduration=2.241282039 podStartE2EDuration="16.198611054s" podCreationTimestamp="2025-10-11 10:56:27 +0000 UTC" firstStartedPulling="2025-10-11 10:56:27.880322021 +0000 UTC m=+1819.854548462" lastFinishedPulling="2025-10-11 10:56:41.837651036 +0000 UTC m=+1833.811877477" observedRunningTime="2025-10-11 10:56:43.189388225 +0000 UTC m=+1835.163614736" watchObservedRunningTime="2025-10-11 10:56:43.198611054 +0000 UTC m=+1835.172837495" Oct 11 10:56:43.350573 master-1 kubenswrapper[4771]: I1011 10:56:43.350523 4771 generic.go:334] "Generic (PLEG): container finished" podID="a23d84be-f5ab-4261-9ba2-d94aaf104a59" containerID="14cbbf6abeb88d28f08a7099ac711df9a488bc85a2f7bd445bc229705a05a25b" exitCode=0 Oct 11 10:56:43.351341 master-1 kubenswrapper[4771]: I1011 10:56:43.350740 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" event={"ID":"a23d84be-f5ab-4261-9ba2-d94aaf104a59","Type":"ContainerDied","Data":"14cbbf6abeb88d28f08a7099ac711df9a488bc85a2f7bd445bc229705a05a25b"} Oct 11 10:56:43.351489 master-1 kubenswrapper[4771]: I1011 10:56:43.351471 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" event={"ID":"a23d84be-f5ab-4261-9ba2-d94aaf104a59","Type":"ContainerStarted","Data":"b0d191f73463f5a71aeb190809caf8100724d2aeec1100c76a864a58130b5a3d"} Oct 11 10:56:43.355456 master-1 kubenswrapper[4771]: I1011 10:56:43.355430 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-conductor-db-sync-tgg64" event={"ID":"5b5e37e3-9afd-4ff3-b992-1e6c28a986ad","Type":"ContainerDied","Data":"8225c71dadd5dd5d7cdb7b603f12129b97565dbfb98b6f8553a5f73b645e62cc"} Oct 11 10:56:43.355526 master-1 kubenswrapper[4771]: I1011 10:56:43.355459 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8225c71dadd5dd5d7cdb7b603f12129b97565dbfb98b6f8553a5f73b645e62cc" Oct 11 10:56:43.358269 master-1 kubenswrapper[4771]: I1011 10:56:43.358202 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-conductor-db-sync-tgg64" Oct 11 10:56:43.974573 master-1 kubenswrapper[4771]: I1011 10:56:43.974514 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/heat-cfnapi-b8cb664c5-5zrqf"] Oct 11 10:56:44.373477 master-1 kubenswrapper[4771]: I1011 10:56:44.373337 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" event={"ID":"a23d84be-f5ab-4261-9ba2-d94aaf104a59","Type":"ContainerStarted","Data":"12c6ff03be76828491f921afc8c9ec6e58880687794d58647b68e34022915241"} Oct 11 10:56:44.374421 master-1 kubenswrapper[4771]: I1011 10:56:44.373584 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:44.551760 master-1 kubenswrapper[4771]: I1011 10:56:44.551672 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/heat-cfnapi-b8cb664c5-5zrqf"] Oct 11 10:56:45.086071 master-1 kubenswrapper[4771]: I1011 10:56:45.085722 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:45.297437 master-1 kubenswrapper[4771]: I1011 10:56:45.297224 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:45.357509 master-1 kubenswrapper[4771]: I1011 10:56:45.357423 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:45.357858 master-1 kubenswrapper[4771]: E1011 10:56:45.357794 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="1fe7833d-9251-4545-ba68-f58c146188f1" containerName="heat-cfnapi" Oct 11 10:56:45.357858 master-1 kubenswrapper[4771]: I1011 10:56:45.357813 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe7833d-9251-4545-ba68-f58c146188f1" containerName="heat-cfnapi" Oct 11 10:56:45.357858 master-1 kubenswrapper[4771]: E1011 10:56:45.357826 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="f85d5cfa-8073-4bbf-9eff-78fde719dadf" containerName="nova-manage" Oct 11 10:56:45.357858 master-1 kubenswrapper[4771]: I1011 10:56:45.357835 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="f85d5cfa-8073-4bbf-9eff-78fde719dadf" containerName="nova-manage" Oct 11 10:56:45.357858 master-1 kubenswrapper[4771]: E1011 10:56:45.357852 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5b5e37e3-9afd-4ff3-b992-1e6c28a986ad" containerName="nova-cell1-conductor-db-sync" Oct 11 10:56:45.357858 master-1 kubenswrapper[4771]: I1011 10:56:45.357860 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5e37e3-9afd-4ff3-b992-1e6c28a986ad" containerName="nova-cell1-conductor-db-sync" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: E1011 10:56:45.357892 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="ceilometer-central-agent" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.357901 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="ceilometer-central-agent" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: E1011 10:56:45.357914 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="sg-core" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.357924 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="sg-core" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: E1011 10:56:45.357941 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="ceilometer-notification-agent" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.357948 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="ceilometer-notification-agent" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: E1011 10:56:45.357961 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="3de492fb-5249-49e2-a327-756234aa92bd" containerName="aodh-db-sync" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.357969 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de492fb-5249-49e2-a327-756234aa92bd" containerName="aodh-db-sync" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: E1011 10:56:45.357984 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="proxy-httpd" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.357991 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="proxy-httpd" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.358151 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="3de492fb-5249-49e2-a327-756234aa92bd" containerName="aodh-db-sync" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.358172 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="proxy-httpd" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.358183 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5b5e37e3-9afd-4ff3-b992-1e6c28a986ad" containerName="nova-cell1-conductor-db-sync" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.358196 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="f85d5cfa-8073-4bbf-9eff-78fde719dadf" containerName="nova-manage" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.358207 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="ceilometer-central-agent" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.358214 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="sg-core" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.358225 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="1fe7833d-9251-4545-ba68-f58c146188f1" containerName="heat-cfnapi" Oct 11 10:56:45.358558 master-1 kubenswrapper[4771]: I1011 10:56:45.358239 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" containerName="ceilometer-notification-agent" Oct 11 10:56:45.360729 master-1 kubenswrapper[4771]: I1011 10:56:45.360282 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:56:45.362903 master-1 kubenswrapper[4771]: W1011 10:56:45.362827 4771 reflector.go:561] object-"openstack"/"ceilometer-config-data": failed to list *v1.Secret: secrets "ceilometer-config-data" is forbidden: User "system:node:master-1" cannot list resource "secrets" in API group "" in the namespace "openstack": no relationship found between node 'master-1' and this object Oct 11 10:56:45.362903 master-1 kubenswrapper[4771]: E1011 10:56:45.362893 4771 reflector.go:158] "Unhandled Error" err="object-\"openstack\"/\"ceilometer-config-data\": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets \"ceilometer-config-data\" is forbidden: User \"system:node:master-1\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openstack\": no relationship found between node 'master-1' and this object" logger="UnhandledError" Oct 11 10:56:45.363199 master-1 kubenswrapper[4771]: W1011 10:56:45.362938 4771 reflector.go:561] object-"openstack"/"cert-ceilometer-internal-svc": failed to list *v1.Secret: secrets "cert-ceilometer-internal-svc" is forbidden: User "system:node:master-1" cannot list resource "secrets" in API group "" in the namespace "openstack": no relationship found between node 'master-1' and this object Oct 11 10:56:45.363199 master-1 kubenswrapper[4771]: E1011 10:56:45.362996 4771 reflector.go:158] "Unhandled Error" err="object-\"openstack\"/\"cert-ceilometer-internal-svc\": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets \"cert-ceilometer-internal-svc\" is forbidden: User \"system:node:master-1\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openstack\": no relationship found between node 'master-1' and this object" logger="UnhandledError" Oct 11 10:56:45.363579 master-1 kubenswrapper[4771]: W1011 10:56:45.363393 4771 reflector.go:561] object-"openstack"/"ceilometer-scripts": failed to list *v1.Secret: secrets "ceilometer-scripts" is forbidden: User "system:node:master-1" cannot list resource "secrets" in API group "" in the namespace "openstack": no relationship found between node 'master-1' and this object Oct 11 10:56:45.363579 master-1 kubenswrapper[4771]: E1011 10:56:45.363450 4771 reflector.go:158] "Unhandled Error" err="object-\"openstack\"/\"ceilometer-scripts\": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets \"ceilometer-scripts\" is forbidden: User \"system:node:master-1\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openstack\": no relationship found between node 'master-1' and this object" logger="UnhandledError" Oct 11 10:56:45.382903 master-1 kubenswrapper[4771]: I1011 10:56:45.382488 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" podStartSLOduration=7.382462321 podStartE2EDuration="7.382462321s" podCreationTimestamp="2025-10-11 10:56:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:56:45.377167287 +0000 UTC m=+1837.351393778" watchObservedRunningTime="2025-10-11 10:56:45.382462321 +0000 UTC m=+1837.356688772" Oct 11 10:56:45.403748 master-1 kubenswrapper[4771]: I1011 10:56:45.403616 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:45.481827 master-1 kubenswrapper[4771]: I1011 10:56:45.481716 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-run-httpd\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.482123 master-1 kubenswrapper[4771]: I1011 10:56:45.481954 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.482123 master-1 kubenswrapper[4771]: I1011 10:56:45.482013 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjb6\" (UniqueName: \"kubernetes.io/projected/e9356157-35da-4cf7-a755-86123f5e09a0-kube-api-access-5wjb6\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.482123 master-1 kubenswrapper[4771]: I1011 10:56:45.482077 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-log-httpd\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.482344 master-1 kubenswrapper[4771]: I1011 10:56:45.482127 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-config-data\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.482344 master-1 kubenswrapper[4771]: I1011 10:56:45.482292 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-scripts\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.482515 master-1 kubenswrapper[4771]: I1011 10:56:45.482452 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.482587 master-1 kubenswrapper[4771]: I1011 10:56:45.482553 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.585007 master-1 kubenswrapper[4771]: I1011 10:56:45.584888 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.585392 master-1 kubenswrapper[4771]: I1011 10:56:45.585030 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.585392 master-1 kubenswrapper[4771]: I1011 10:56:45.585122 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-run-httpd\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.585392 master-1 kubenswrapper[4771]: I1011 10:56:45.585282 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.585392 master-1 kubenswrapper[4771]: I1011 10:56:45.585343 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjb6\" (UniqueName: \"kubernetes.io/projected/e9356157-35da-4cf7-a755-86123f5e09a0-kube-api-access-5wjb6\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.585801 master-1 kubenswrapper[4771]: I1011 10:56:45.585470 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-log-httpd\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.585934 master-1 kubenswrapper[4771]: I1011 10:56:45.585837 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-run-httpd\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.586267 master-1 kubenswrapper[4771]: I1011 10:56:45.586224 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-log-httpd\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.586443 master-1 kubenswrapper[4771]: I1011 10:56:45.586313 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-config-data\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.586641 master-1 kubenswrapper[4771]: I1011 10:56:45.586586 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-scripts\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.589722 master-1 kubenswrapper[4771]: I1011 10:56:45.589655 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:45.609608 master-1 kubenswrapper[4771]: I1011 10:56:45.609515 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjb6\" (UniqueName: \"kubernetes.io/projected/e9356157-35da-4cf7-a755-86123f5e09a0-kube-api-access-5wjb6\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:46.184243 master-1 kubenswrapper[4771]: I1011 10:56:46.184145 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ceilometer-internal-svc" Oct 11 10:56:46.190707 master-1 kubenswrapper[4771]: I1011 10:56:46.190612 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:46.367245 master-1 kubenswrapper[4771]: I1011 10:56:46.367132 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 10:56:46.375820 master-1 kubenswrapper[4771]: I1011 10:56:46.375718 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-scripts\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:46.455155 master-1 kubenswrapper[4771]: I1011 10:56:46.454929 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe7833d-9251-4545-ba68-f58c146188f1" path="/var/lib/kubelet/pods/1fe7833d-9251-4545-ba68-f58c146188f1/volumes" Oct 11 10:56:46.456471 master-1 kubenswrapper[4771]: I1011 10:56:46.456421 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0cc5394-b33f-41a9-bbe2-d772e75a8f58" path="/var/lib/kubelet/pods/d0cc5394-b33f-41a9-bbe2-d772e75a8f58/volumes" Oct 11 10:56:46.533416 master-1 kubenswrapper[4771]: I1011 10:56:46.533323 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 10:56:46.542558 master-1 kubenswrapper[4771]: I1011 10:56:46.542486 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:46.548297 master-1 kubenswrapper[4771]: I1011 10:56:46.548211 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-config-data\") pod \"ceilometer-0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " pod="openstack/ceilometer-0" Oct 11 10:56:46.589990 master-1 kubenswrapper[4771]: I1011 10:56:46.589915 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:56:47.306747 master-1 kubenswrapper[4771]: I1011 10:56:47.306637 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell1-conductor-0"] Oct 11 10:56:47.308610 master-1 kubenswrapper[4771]: I1011 10:56:47.308559 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:47.312053 master-1 kubenswrapper[4771]: I1011 10:56:47.311990 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell1-conductor-config-data" Oct 11 10:56:47.393273 master-1 kubenswrapper[4771]: I1011 10:56:47.393194 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-conductor-0"] Oct 11 10:56:47.430549 master-1 kubenswrapper[4771]: I1011 10:56:47.430487 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/b5934858-6421-4f73-9a74-3111541cc898-config-data\") pod \"nova-cell1-conductor-0\" (UID: \"b5934858-6421-4f73-9a74-3111541cc898\") " pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:47.430783 master-1 kubenswrapper[4771]: I1011 10:56:47.430661 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b5934858-6421-4f73-9a74-3111541cc898-combined-ca-bundle\") pod \"nova-cell1-conductor-0\" (UID: \"b5934858-6421-4f73-9a74-3111541cc898\") " pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:47.430783 master-1 kubenswrapper[4771]: I1011 10:56:47.430717 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrm8\" (UniqueName: \"kubernetes.io/projected/b5934858-6421-4f73-9a74-3111541cc898-kube-api-access-mcrm8\") pod \"nova-cell1-conductor-0\" (UID: \"b5934858-6421-4f73-9a74-3111541cc898\") " pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:47.531321 master-1 kubenswrapper[4771]: I1011 10:56:47.531227 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:47.532790 master-1 kubenswrapper[4771]: I1011 10:56:47.532721 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b5934858-6421-4f73-9a74-3111541cc898-combined-ca-bundle\") pod \"nova-cell1-conductor-0\" (UID: \"b5934858-6421-4f73-9a74-3111541cc898\") " pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:47.532924 master-1 kubenswrapper[4771]: I1011 10:56:47.532819 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrm8\" (UniqueName: \"kubernetes.io/projected/b5934858-6421-4f73-9a74-3111541cc898-kube-api-access-mcrm8\") pod \"nova-cell1-conductor-0\" (UID: \"b5934858-6421-4f73-9a74-3111541cc898\") " pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:47.533011 master-1 kubenswrapper[4771]: I1011 10:56:47.532944 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/b5934858-6421-4f73-9a74-3111541cc898-config-data\") pod \"nova-cell1-conductor-0\" (UID: \"b5934858-6421-4f73-9a74-3111541cc898\") " pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:47.538443 master-1 kubenswrapper[4771]: I1011 10:56:47.538310 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/b5934858-6421-4f73-9a74-3111541cc898-config-data\") pod \"nova-cell1-conductor-0\" (UID: \"b5934858-6421-4f73-9a74-3111541cc898\") " pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:47.541444 master-1 kubenswrapper[4771]: I1011 10:56:47.541381 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b5934858-6421-4f73-9a74-3111541cc898-combined-ca-bundle\") pod \"nova-cell1-conductor-0\" (UID: \"b5934858-6421-4f73-9a74-3111541cc898\") " pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:47.554868 master-1 kubenswrapper[4771]: I1011 10:56:47.554801 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-api-0" Oct 11 10:56:47.556611 master-1 kubenswrapper[4771]: I1011 10:56:47.556512 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-api-0" Oct 11 10:56:47.557881 master-1 kubenswrapper[4771]: I1011 10:56:47.557785 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-api-0" Oct 11 10:56:47.559865 master-1 kubenswrapper[4771]: I1011 10:56:47.559810 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-api-0" Oct 11 10:56:47.964726 master-1 kubenswrapper[4771]: I1011 10:56:47.964552 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-metadata-1" Oct 11 10:56:47.965140 master-1 kubenswrapper[4771]: I1011 10:56:47.965074 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-metadata-1" Oct 11 10:56:47.966808 master-1 kubenswrapper[4771]: I1011 10:56:47.966754 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-metadata-1" Oct 11 10:56:48.420060 master-1 kubenswrapper[4771]: I1011 10:56:48.419993 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrm8\" (UniqueName: \"kubernetes.io/projected/b5934858-6421-4f73-9a74-3111541cc898-kube-api-access-mcrm8\") pod \"nova-cell1-conductor-0\" (UID: \"b5934858-6421-4f73-9a74-3111541cc898\") " pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:48.436454 master-1 kubenswrapper[4771]: I1011 10:56:48.433444 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerStarted","Data":"4a07614606065916a0a6a5a0a24779d53f74c10d0d6a9319c0eda823b68adb65"} Oct 11 10:56:48.436454 master-1 kubenswrapper[4771]: I1011 10:56:48.434042 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-api-0" Oct 11 10:56:48.450431 master-1 kubenswrapper[4771]: I1011 10:56:48.450314 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-api-0" Oct 11 10:56:48.450431 master-1 kubenswrapper[4771]: I1011 10:56:48.450402 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-metadata-1" Oct 11 10:56:48.538129 master-1 kubenswrapper[4771]: I1011 10:56:48.538027 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:48.619825 master-1 kubenswrapper[4771]: I1011 10:56:48.619739 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:56:49.441848 master-1 kubenswrapper[4771]: I1011 10:56:49.441547 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-conductor-0"] Oct 11 10:56:49.463896 master-1 kubenswrapper[4771]: I1011 10:56:49.463702 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-conductor-0" event={"ID":"b5934858-6421-4f73-9a74-3111541cc898","Type":"ContainerStarted","Data":"a03501b6649e0c6f0e4295eaab4fcf27313def1221099088aa148210423ed3da"} Oct 11 10:56:49.465447 master-1 kubenswrapper[4771]: I1011 10:56:49.465386 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerStarted","Data":"5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a"} Oct 11 10:56:49.572112 master-1 kubenswrapper[4771]: I1011 10:56:49.572054 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-768f954cfc-gvj8j"] Oct 11 10:56:49.572574 master-1 kubenswrapper[4771]: I1011 10:56:49.572394 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" podUID="7f2f3d22-d709-4602-bb25-2c17626b75f1" containerName="dnsmasq-dns" containerID="cri-o://f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973" gracePeriod=10 Oct 11 10:56:50.078936 master-1 kubenswrapper[4771]: I1011 10:56:50.078864 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:56:50.199557 master-1 kubenswrapper[4771]: I1011 10:56:50.199486 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-swift-storage-0\") pod \"7f2f3d22-d709-4602-bb25-2c17626b75f1\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " Oct 11 10:56:50.199933 master-1 kubenswrapper[4771]: I1011 10:56:50.199637 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64zxc\" (UniqueName: \"kubernetes.io/projected/7f2f3d22-d709-4602-bb25-2c17626b75f1-kube-api-access-64zxc\") pod \"7f2f3d22-d709-4602-bb25-2c17626b75f1\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " Oct 11 10:56:50.199933 master-1 kubenswrapper[4771]: I1011 10:56:50.199683 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-nb\") pod \"7f2f3d22-d709-4602-bb25-2c17626b75f1\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " Oct 11 10:56:50.199933 master-1 kubenswrapper[4771]: I1011 10:56:50.199735 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-svc\") pod \"7f2f3d22-d709-4602-bb25-2c17626b75f1\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " Oct 11 10:56:50.199933 master-1 kubenswrapper[4771]: I1011 10:56:50.199808 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-config\") pod \"7f2f3d22-d709-4602-bb25-2c17626b75f1\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " Oct 11 10:56:50.199933 master-1 kubenswrapper[4771]: I1011 10:56:50.199858 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-sb\") pod \"7f2f3d22-d709-4602-bb25-2c17626b75f1\" (UID: \"7f2f3d22-d709-4602-bb25-2c17626b75f1\") " Oct 11 10:56:50.206793 master-1 kubenswrapper[4771]: I1011 10:56:50.206712 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2f3d22-d709-4602-bb25-2c17626b75f1-kube-api-access-64zxc" (OuterVolumeSpecName: "kube-api-access-64zxc") pod "7f2f3d22-d709-4602-bb25-2c17626b75f1" (UID: "7f2f3d22-d709-4602-bb25-2c17626b75f1"). InnerVolumeSpecName "kube-api-access-64zxc". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:50.239242 master-1 kubenswrapper[4771]: I1011 10:56:50.239003 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-config" (OuterVolumeSpecName: "config") pod "7f2f3d22-d709-4602-bb25-2c17626b75f1" (UID: "7f2f3d22-d709-4602-bb25-2c17626b75f1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:56:50.247027 master-1 kubenswrapper[4771]: I1011 10:56:50.246960 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-swift-storage-0" (OuterVolumeSpecName: "dns-swift-storage-0") pod "7f2f3d22-d709-4602-bb25-2c17626b75f1" (UID: "7f2f3d22-d709-4602-bb25-2c17626b75f1"). InnerVolumeSpecName "dns-swift-storage-0". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:56:50.262203 master-1 kubenswrapper[4771]: I1011 10:56:50.261879 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-nb" (OuterVolumeSpecName: "ovsdbserver-nb") pod "7f2f3d22-d709-4602-bb25-2c17626b75f1" (UID: "7f2f3d22-d709-4602-bb25-2c17626b75f1"). InnerVolumeSpecName "ovsdbserver-nb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:56:50.262453 master-1 kubenswrapper[4771]: I1011 10:56:50.262215 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-sb" (OuterVolumeSpecName: "ovsdbserver-sb") pod "7f2f3d22-d709-4602-bb25-2c17626b75f1" (UID: "7f2f3d22-d709-4602-bb25-2c17626b75f1"). InnerVolumeSpecName "ovsdbserver-sb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:56:50.273444 master-1 kubenswrapper[4771]: I1011 10:56:50.273378 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "7f2f3d22-d709-4602-bb25-2c17626b75f1" (UID: "7f2f3d22-d709-4602-bb25-2c17626b75f1"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:56:50.302837 master-1 kubenswrapper[4771]: I1011 10:56:50.302781 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:50.302837 master-1 kubenswrapper[4771]: I1011 10:56:50.302823 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-sb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:50.303075 master-1 kubenswrapper[4771]: I1011 10:56:50.302866 4771 reconciler_common.go:293] "Volume detached for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-swift-storage-0\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:50.303075 master-1 kubenswrapper[4771]: I1011 10:56:50.302878 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-64zxc\" (UniqueName: \"kubernetes.io/projected/7f2f3d22-d709-4602-bb25-2c17626b75f1-kube-api-access-64zxc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:50.303075 master-1 kubenswrapper[4771]: I1011 10:56:50.302886 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-ovsdbserver-nb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:50.303075 master-1 kubenswrapper[4771]: I1011 10:56:50.302895 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/7f2f3d22-d709-4602-bb25-2c17626b75f1-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:50.476672 master-1 kubenswrapper[4771]: I1011 10:56:50.476608 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerStarted","Data":"40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b"} Oct 11 10:56:50.479553 master-1 kubenswrapper[4771]: I1011 10:56:50.479436 4771 generic.go:334] "Generic (PLEG): container finished" podID="7f2f3d22-d709-4602-bb25-2c17626b75f1" containerID="f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973" exitCode=0 Oct 11 10:56:50.479553 master-1 kubenswrapper[4771]: I1011 10:56:50.479501 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" event={"ID":"7f2f3d22-d709-4602-bb25-2c17626b75f1","Type":"ContainerDied","Data":"f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973"} Oct 11 10:56:50.479684 master-1 kubenswrapper[4771]: I1011 10:56:50.479593 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" event={"ID":"7f2f3d22-d709-4602-bb25-2c17626b75f1","Type":"ContainerDied","Data":"44d3f4f2f792c76343fbbdc36dc9dcbfc23552c6a12758c872ba3f43c891c162"} Oct 11 10:56:50.479684 master-1 kubenswrapper[4771]: I1011 10:56:50.479626 4771 scope.go:117] "RemoveContainer" containerID="f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973" Oct 11 10:56:50.479684 master-1 kubenswrapper[4771]: I1011 10:56:50.479531 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-768f954cfc-gvj8j" Oct 11 10:56:50.481971 master-1 kubenswrapper[4771]: I1011 10:56:50.481904 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-conductor-0" event={"ID":"b5934858-6421-4f73-9a74-3111541cc898","Type":"ContainerStarted","Data":"9910b1f3b104026b038b4ca3a28b0b7ef93f104d170c8aa8aecb636462fb76f3"} Oct 11 10:56:50.482229 master-1 kubenswrapper[4771]: I1011 10:56:50.482177 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:50.514222 master-1 kubenswrapper[4771]: I1011 10:56:50.514139 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell1-conductor-0" podStartSLOduration=4.514117184 podStartE2EDuration="4.514117184s" podCreationTimestamp="2025-10-11 10:56:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:56:50.508803839 +0000 UTC m=+1842.483030270" watchObservedRunningTime="2025-10-11 10:56:50.514117184 +0000 UTC m=+1842.488343635" Oct 11 10:56:50.549455 master-1 kubenswrapper[4771]: I1011 10:56:50.549348 4771 scope.go:117] "RemoveContainer" containerID="4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1" Oct 11 10:56:50.566777 master-1 kubenswrapper[4771]: I1011 10:56:50.566510 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-768f954cfc-gvj8j"] Oct 11 10:56:50.572955 master-1 kubenswrapper[4771]: I1011 10:56:50.572894 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-768f954cfc-gvj8j"] Oct 11 10:56:50.591244 master-1 kubenswrapper[4771]: I1011 10:56:50.591205 4771 scope.go:117] "RemoveContainer" containerID="f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973" Oct 11 10:56:50.592082 master-1 kubenswrapper[4771]: E1011 10:56:50.591973 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973\": container with ID starting with f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973 not found: ID does not exist" containerID="f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973" Oct 11 10:56:50.592167 master-1 kubenswrapper[4771]: I1011 10:56:50.592105 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973"} err="failed to get container status \"f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973\": rpc error: code = NotFound desc = could not find container \"f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973\": container with ID starting with f842da8207799702100719e16d6957042c6a44764caa5bd18413f4806a961973 not found: ID does not exist" Oct 11 10:56:50.592167 master-1 kubenswrapper[4771]: I1011 10:56:50.592155 4771 scope.go:117] "RemoveContainer" containerID="4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1" Oct 11 10:56:50.592826 master-1 kubenswrapper[4771]: E1011 10:56:50.592797 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1\": container with ID starting with 4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1 not found: ID does not exist" containerID="4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1" Oct 11 10:56:50.593025 master-1 kubenswrapper[4771]: I1011 10:56:50.592990 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1"} err="failed to get container status \"4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1\": rpc error: code = NotFound desc = could not find container \"4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1\": container with ID starting with 4fae999cde829daae9a63e594b851b24a38e90a61f567ae366b32cc609a54cf1 not found: ID does not exist" Oct 11 10:56:51.364193 master-1 kubenswrapper[4771]: I1011 10:56:51.364063 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:51.495130 master-1 kubenswrapper[4771]: I1011 10:56:51.495046 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerStarted","Data":"815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961"} Oct 11 10:56:52.450077 master-1 kubenswrapper[4771]: I1011 10:56:52.450024 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f2f3d22-d709-4602-bb25-2c17626b75f1" path="/var/lib/kubelet/pods/7f2f3d22-d709-4602-bb25-2c17626b75f1/volumes" Oct 11 10:56:52.503113 master-1 kubenswrapper[4771]: I1011 10:56:52.503044 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerStarted","Data":"32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010"} Oct 11 10:56:52.503331 master-1 kubenswrapper[4771]: I1011 10:56:52.503239 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="ceilometer-central-agent" containerID="cri-o://5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a" gracePeriod=30 Oct 11 10:56:52.503546 master-1 kubenswrapper[4771]: I1011 10:56:52.503520 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 10:56:52.503764 master-1 kubenswrapper[4771]: I1011 10:56:52.503737 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="proxy-httpd" containerID="cri-o://32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010" gracePeriod=30 Oct 11 10:56:52.503813 master-1 kubenswrapper[4771]: I1011 10:56:52.503796 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="sg-core" containerID="cri-o://815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961" gracePeriod=30 Oct 11 10:56:52.503861 master-1 kubenswrapper[4771]: I1011 10:56:52.503842 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="ceilometer-notification-agent" containerID="cri-o://40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b" gracePeriod=30 Oct 11 10:56:52.539657 master-1 kubenswrapper[4771]: I1011 10:56:52.539547 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=3.12288801 podStartE2EDuration="7.539521911s" podCreationTimestamp="2025-10-11 10:56:45 +0000 UTC" firstStartedPulling="2025-10-11 10:56:47.459936986 +0000 UTC m=+1839.434163437" lastFinishedPulling="2025-10-11 10:56:51.876570877 +0000 UTC m=+1843.850797338" observedRunningTime="2025-10-11 10:56:52.530867178 +0000 UTC m=+1844.505093619" watchObservedRunningTime="2025-10-11 10:56:52.539521911 +0000 UTC m=+1844.513748352" Oct 11 10:56:53.519012 master-1 kubenswrapper[4771]: I1011 10:56:53.518939 4771 generic.go:334] "Generic (PLEG): container finished" podID="e9356157-35da-4cf7-a755-86123f5e09a0" containerID="32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010" exitCode=0 Oct 11 10:56:53.519012 master-1 kubenswrapper[4771]: I1011 10:56:53.518981 4771 generic.go:334] "Generic (PLEG): container finished" podID="e9356157-35da-4cf7-a755-86123f5e09a0" containerID="815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961" exitCode=2 Oct 11 10:56:53.519012 master-1 kubenswrapper[4771]: I1011 10:56:53.518991 4771 generic.go:334] "Generic (PLEG): container finished" podID="e9356157-35da-4cf7-a755-86123f5e09a0" containerID="40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b" exitCode=0 Oct 11 10:56:53.519012 master-1 kubenswrapper[4771]: I1011 10:56:53.519017 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerDied","Data":"32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010"} Oct 11 10:56:53.520054 master-1 kubenswrapper[4771]: I1011 10:56:53.519050 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerDied","Data":"815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961"} Oct 11 10:56:53.520054 master-1 kubenswrapper[4771]: I1011 10:56:53.519068 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerDied","Data":"40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b"} Oct 11 10:56:58.211881 master-1 kubenswrapper[4771]: I1011 10:56:58.211807 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.290100 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-run-httpd\") pod \"e9356157-35da-4cf7-a755-86123f5e09a0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.290194 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-combined-ca-bundle\") pod \"e9356157-35da-4cf7-a755-86123f5e09a0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.290261 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-ceilometer-tls-certs\") pod \"e9356157-35da-4cf7-a755-86123f5e09a0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.290407 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wjb6\" (UniqueName: \"kubernetes.io/projected/e9356157-35da-4cf7-a755-86123f5e09a0-kube-api-access-5wjb6\") pod \"e9356157-35da-4cf7-a755-86123f5e09a0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.290441 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-config-data\") pod \"e9356157-35da-4cf7-a755-86123f5e09a0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.290487 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-scripts\") pod \"e9356157-35da-4cf7-a755-86123f5e09a0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.290531 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-log-httpd\") pod \"e9356157-35da-4cf7-a755-86123f5e09a0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.290571 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-sg-core-conf-yaml\") pod \"e9356157-35da-4cf7-a755-86123f5e09a0\" (UID: \"e9356157-35da-4cf7-a755-86123f5e09a0\") " Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.290696 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "e9356157-35da-4cf7-a755-86123f5e09a0" (UID: "e9356157-35da-4cf7-a755-86123f5e09a0"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:58.291307 master-1 kubenswrapper[4771]: I1011 10:56:58.291053 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:58.296963 master-1 kubenswrapper[4771]: I1011 10:56:58.296894 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "e9356157-35da-4cf7-a755-86123f5e09a0" (UID: "e9356157-35da-4cf7-a755-86123f5e09a0"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:56:58.298483 master-1 kubenswrapper[4771]: I1011 10:56:58.298394 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-scripts" (OuterVolumeSpecName: "scripts") pod "e9356157-35da-4cf7-a755-86123f5e09a0" (UID: "e9356157-35da-4cf7-a755-86123f5e09a0"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:58.299097 master-1 kubenswrapper[4771]: I1011 10:56:58.298987 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9356157-35da-4cf7-a755-86123f5e09a0-kube-api-access-5wjb6" (OuterVolumeSpecName: "kube-api-access-5wjb6") pod "e9356157-35da-4cf7-a755-86123f5e09a0" (UID: "e9356157-35da-4cf7-a755-86123f5e09a0"). InnerVolumeSpecName "kube-api-access-5wjb6". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:56:58.325092 master-1 kubenswrapper[4771]: I1011 10:56:58.325009 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "e9356157-35da-4cf7-a755-86123f5e09a0" (UID: "e9356157-35da-4cf7-a755-86123f5e09a0"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:58.338195 master-1 kubenswrapper[4771]: I1011 10:56:58.338134 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-ceilometer-tls-certs" (OuterVolumeSpecName: "ceilometer-tls-certs") pod "e9356157-35da-4cf7-a755-86123f5e09a0" (UID: "e9356157-35da-4cf7-a755-86123f5e09a0"). InnerVolumeSpecName "ceilometer-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:58.368553 master-1 kubenswrapper[4771]: I1011 10:56:58.368468 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "e9356157-35da-4cf7-a755-86123f5e09a0" (UID: "e9356157-35da-4cf7-a755-86123f5e09a0"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:58.392914 master-1 kubenswrapper[4771]: I1011 10:56:58.392756 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-5wjb6\" (UniqueName: \"kubernetes.io/projected/e9356157-35da-4cf7-a755-86123f5e09a0-kube-api-access-5wjb6\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:58.392914 master-1 kubenswrapper[4771]: I1011 10:56:58.392790 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:58.392914 master-1 kubenswrapper[4771]: I1011 10:56:58.392806 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e9356157-35da-4cf7-a755-86123f5e09a0-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:58.392914 master-1 kubenswrapper[4771]: I1011 10:56:58.392815 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:58.392914 master-1 kubenswrapper[4771]: I1011 10:56:58.392824 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:58.392914 master-1 kubenswrapper[4771]: I1011 10:56:58.392833 4771 reconciler_common.go:293] "Volume detached for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-ceilometer-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:58.399733 master-1 kubenswrapper[4771]: I1011 10:56:58.399667 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-config-data" (OuterVolumeSpecName: "config-data") pod "e9356157-35da-4cf7-a755-86123f5e09a0" (UID: "e9356157-35da-4cf7-a755-86123f5e09a0"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:56:58.494570 master-1 kubenswrapper[4771]: I1011 10:56:58.494519 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e9356157-35da-4cf7-a755-86123f5e09a0-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:56:58.580563 master-1 kubenswrapper[4771]: I1011 10:56:58.580210 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-cell1-conductor-0" Oct 11 10:56:58.594541 master-1 kubenswrapper[4771]: I1011 10:56:58.594465 4771 generic.go:334] "Generic (PLEG): container finished" podID="e9356157-35da-4cf7-a755-86123f5e09a0" containerID="5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a" exitCode=0 Oct 11 10:56:58.594541 master-1 kubenswrapper[4771]: I1011 10:56:58.594542 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerDied","Data":"5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a"} Oct 11 10:56:58.594857 master-1 kubenswrapper[4771]: I1011 10:56:58.594589 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e9356157-35da-4cf7-a755-86123f5e09a0","Type":"ContainerDied","Data":"4a07614606065916a0a6a5a0a24779d53f74c10d0d6a9319c0eda823b68adb65"} Oct 11 10:56:58.594857 master-1 kubenswrapper[4771]: I1011 10:56:58.594613 4771 scope.go:117] "RemoveContainer" containerID="32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010" Oct 11 10:56:58.594857 master-1 kubenswrapper[4771]: I1011 10:56:58.594658 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:56:58.634274 master-1 kubenswrapper[4771]: I1011 10:56:58.634209 4771 scope.go:117] "RemoveContainer" containerID="815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961" Oct 11 10:56:58.639440 master-1 kubenswrapper[4771]: I1011 10:56:58.639389 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:58.657897 master-1 kubenswrapper[4771]: I1011 10:56:58.657824 4771 scope.go:117] "RemoveContainer" containerID="40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b" Oct 11 10:56:58.669085 master-1 kubenswrapper[4771]: I1011 10:56:58.669020 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:58.676813 master-1 kubenswrapper[4771]: I1011 10:56:58.676770 4771 scope.go:117] "RemoveContainer" containerID="5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a" Oct 11 10:56:58.700954 master-1 kubenswrapper[4771]: I1011 10:56:58.700895 4771 scope.go:117] "RemoveContainer" containerID="32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010" Oct 11 10:56:58.701426 master-1 kubenswrapper[4771]: E1011 10:56:58.701315 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010\": container with ID starting with 32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010 not found: ID does not exist" containerID="32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010" Oct 11 10:56:58.701426 master-1 kubenswrapper[4771]: I1011 10:56:58.701391 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010"} err="failed to get container status \"32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010\": rpc error: code = NotFound desc = could not find container \"32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010\": container with ID starting with 32a9f5966292aad5b339211592f59adf619a4421ecf5cf67f9fe01db264f2010 not found: ID does not exist" Oct 11 10:56:58.701569 master-1 kubenswrapper[4771]: I1011 10:56:58.701432 4771 scope.go:117] "RemoveContainer" containerID="815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961" Oct 11 10:56:58.701973 master-1 kubenswrapper[4771]: E1011 10:56:58.701909 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961\": container with ID starting with 815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961 not found: ID does not exist" containerID="815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961" Oct 11 10:56:58.702069 master-1 kubenswrapper[4771]: I1011 10:56:58.701971 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961"} err="failed to get container status \"815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961\": rpc error: code = NotFound desc = could not find container \"815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961\": container with ID starting with 815d31b7bfb4da3463e478b429dc12c791555c18207b5e1a4435119f5ff42961 not found: ID does not exist" Oct 11 10:56:58.702069 master-1 kubenswrapper[4771]: I1011 10:56:58.701990 4771 scope.go:117] "RemoveContainer" containerID="40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b" Oct 11 10:56:58.702636 master-1 kubenswrapper[4771]: E1011 10:56:58.702577 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b\": container with ID starting with 40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b not found: ID does not exist" containerID="40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b" Oct 11 10:56:58.702636 master-1 kubenswrapper[4771]: I1011 10:56:58.702615 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b"} err="failed to get container status \"40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b\": rpc error: code = NotFound desc = could not find container \"40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b\": container with ID starting with 40e6f1404086cf0d3b630377374f5782adbaa5371b9d23d9249ebbb3d545f95b not found: ID does not exist" Oct 11 10:56:58.702636 master-1 kubenswrapper[4771]: I1011 10:56:58.702639 4771 scope.go:117] "RemoveContainer" containerID="5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a" Oct 11 10:56:58.703178 master-1 kubenswrapper[4771]: E1011 10:56:58.703149 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a\": container with ID starting with 5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a not found: ID does not exist" containerID="5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a" Oct 11 10:56:58.703178 master-1 kubenswrapper[4771]: I1011 10:56:58.703175 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a"} err="failed to get container status \"5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a\": rpc error: code = NotFound desc = could not find container \"5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a\": container with ID starting with 5d78c693491970b1a8eb3619842707db804737dd9e6aea89c13ff6875afaf87a not found: ID does not exist" Oct 11 10:56:58.848928 master-1 kubenswrapper[4771]: I1011 10:56:58.848823 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:58.849245 master-1 kubenswrapper[4771]: E1011 10:56:58.849187 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="ceilometer-central-agent" Oct 11 10:56:58.849245 master-1 kubenswrapper[4771]: I1011 10:56:58.849204 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="ceilometer-central-agent" Oct 11 10:56:58.849245 master-1 kubenswrapper[4771]: E1011 10:56:58.849227 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="sg-core" Oct 11 10:56:58.849245 master-1 kubenswrapper[4771]: I1011 10:56:58.849236 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="sg-core" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: E1011 10:56:58.849262 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="proxy-httpd" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: I1011 10:56:58.849273 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="proxy-httpd" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: E1011 10:56:58.849291 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="7f2f3d22-d709-4602-bb25-2c17626b75f1" containerName="dnsmasq-dns" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: I1011 10:56:58.849300 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2f3d22-d709-4602-bb25-2c17626b75f1" containerName="dnsmasq-dns" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: E1011 10:56:58.849315 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="7f2f3d22-d709-4602-bb25-2c17626b75f1" containerName="init" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: I1011 10:56:58.849324 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2f3d22-d709-4602-bb25-2c17626b75f1" containerName="init" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: E1011 10:56:58.849336 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="ceilometer-notification-agent" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: I1011 10:56:58.849346 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="ceilometer-notification-agent" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: I1011 10:56:58.849535 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="proxy-httpd" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: I1011 10:56:58.849563 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="sg-core" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: I1011 10:56:58.849583 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="ceilometer-notification-agent" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: I1011 10:56:58.849596 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="7f2f3d22-d709-4602-bb25-2c17626b75f1" containerName="dnsmasq-dns" Oct 11 10:56:58.849618 master-1 kubenswrapper[4771]: I1011 10:56:58.849616 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" containerName="ceilometer-central-agent" Oct 11 10:56:58.851594 master-1 kubenswrapper[4771]: I1011 10:56:58.851548 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:56:58.853894 master-1 kubenswrapper[4771]: W1011 10:56:58.853826 4771 reflector.go:561] object-"openstack"/"ceilometer-config-data": failed to list *v1.Secret: secrets "ceilometer-config-data" is forbidden: User "system:node:master-1" cannot list resource "secrets" in API group "" in the namespace "openstack": no relationship found between node 'master-1' and this object Oct 11 10:56:58.853894 master-1 kubenswrapper[4771]: E1011 10:56:58.853870 4771 reflector.go:158] "Unhandled Error" err="object-\"openstack\"/\"ceilometer-config-data\": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets \"ceilometer-config-data\" is forbidden: User \"system:node:master-1\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openstack\": no relationship found between node 'master-1' and this object" logger="UnhandledError" Oct 11 10:56:58.854433 master-1 kubenswrapper[4771]: W1011 10:56:58.854306 4771 reflector.go:561] object-"openstack"/"ceilometer-scripts": failed to list *v1.Secret: secrets "ceilometer-scripts" is forbidden: User "system:node:master-1" cannot list resource "secrets" in API group "" in the namespace "openstack": no relationship found between node 'master-1' and this object Oct 11 10:56:58.854433 master-1 kubenswrapper[4771]: E1011 10:56:58.854336 4771 reflector.go:158] "Unhandled Error" err="object-\"openstack\"/\"ceilometer-scripts\": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets \"ceilometer-scripts\" is forbidden: User \"system:node:master-1\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openstack\": no relationship found between node 'master-1' and this object" logger="UnhandledError" Oct 11 10:56:58.854708 master-1 kubenswrapper[4771]: W1011 10:56:58.854471 4771 reflector.go:561] object-"openstack"/"cert-ceilometer-internal-svc": failed to list *v1.Secret: secrets "cert-ceilometer-internal-svc" is forbidden: User "system:node:master-1" cannot list resource "secrets" in API group "" in the namespace "openstack": no relationship found between node 'master-1' and this object Oct 11 10:56:58.854708 master-1 kubenswrapper[4771]: E1011 10:56:58.854491 4771 reflector.go:158] "Unhandled Error" err="object-\"openstack\"/\"cert-ceilometer-internal-svc\": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets \"cert-ceilometer-internal-svc\" is forbidden: User \"system:node:master-1\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openstack\": no relationship found between node 'master-1' and this object" logger="UnhandledError" Oct 11 10:56:59.005265 master-1 kubenswrapper[4771]: I1011 10:56:59.005080 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.005631 master-1 kubenswrapper[4771]: I1011 10:56:59.005448 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-config-data\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.005631 master-1 kubenswrapper[4771]: I1011 10:56:59.005534 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-run-httpd\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.005631 master-1 kubenswrapper[4771]: I1011 10:56:59.005606 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-log-httpd\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.005886 master-1 kubenswrapper[4771]: I1011 10:56:59.005694 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.005966 master-1 kubenswrapper[4771]: I1011 10:56:59.005911 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.006187 master-1 kubenswrapper[4771]: I1011 10:56:59.006024 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtfvl\" (UniqueName: \"kubernetes.io/projected/9a279d25-518e-4a12-8b75-e3781fb22f05-kube-api-access-wtfvl\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.006187 master-1 kubenswrapper[4771]: I1011 10:56:59.006140 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-scripts\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.039465 master-1 kubenswrapper[4771]: I1011 10:56:59.039341 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:56:59.108617 master-1 kubenswrapper[4771]: I1011 10:56:59.108489 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-scripts\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.108617 master-1 kubenswrapper[4771]: I1011 10:56:59.108607 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.109091 master-1 kubenswrapper[4771]: I1011 10:56:59.108813 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-config-data\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.109091 master-1 kubenswrapper[4771]: I1011 10:56:59.108862 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-run-httpd\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.109091 master-1 kubenswrapper[4771]: I1011 10:56:59.108915 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-log-httpd\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.109091 master-1 kubenswrapper[4771]: I1011 10:56:59.108955 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.109091 master-1 kubenswrapper[4771]: I1011 10:56:59.109034 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.109705 master-1 kubenswrapper[4771]: I1011 10:56:59.109117 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-wtfvl\" (UniqueName: \"kubernetes.io/projected/9a279d25-518e-4a12-8b75-e3781fb22f05-kube-api-access-wtfvl\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.109705 master-1 kubenswrapper[4771]: I1011 10:56:59.109650 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-log-httpd\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.109922 master-1 kubenswrapper[4771]: I1011 10:56:59.109755 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-run-httpd\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.113303 master-1 kubenswrapper[4771]: I1011 10:56:59.113222 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:56:59.138387 master-1 kubenswrapper[4771]: I1011 10:56:59.138278 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtfvl\" (UniqueName: \"kubernetes.io/projected/9a279d25-518e-4a12-8b75-e3781fb22f05-kube-api-access-wtfvl\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:57:00.059488 master-1 kubenswrapper[4771]: I1011 10:57:00.059407 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ceilometer-internal-svc" Oct 11 10:57:00.065497 master-1 kubenswrapper[4771]: I1011 10:57:00.065443 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:57:00.083640 master-1 kubenswrapper[4771]: I1011 10:57:00.083605 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 10:57:00.093122 master-1 kubenswrapper[4771]: I1011 10:57:00.093097 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-config-data\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:57:00.094679 master-1 kubenswrapper[4771]: I1011 10:57:00.094628 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:57:00.110479 master-1 kubenswrapper[4771]: E1011 10:57:00.108878 4771 secret.go:189] Couldn't get secret openstack/ceilometer-scripts: failed to sync secret cache: timed out waiting for the condition Oct 11 10:57:00.110479 master-1 kubenswrapper[4771]: E1011 10:57:00.108977 4771 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-scripts podName:9a279d25-518e-4a12-8b75-e3781fb22f05 nodeName:}" failed. No retries permitted until 2025-10-11 10:57:00.608950395 +0000 UTC m=+1852.583176866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "scripts" (UniqueName: "kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-scripts") pod "ceilometer-0" (UID: "9a279d25-518e-4a12-8b75-e3781fb22f05") : failed to sync secret cache: timed out waiting for the condition Oct 11 10:57:00.300331 master-1 kubenswrapper[4771]: I1011 10:57:00.300258 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 10:57:00.455873 master-1 kubenswrapper[4771]: I1011 10:57:00.455674 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9356157-35da-4cf7-a755-86123f5e09a0" path="/var/lib/kubelet/pods/e9356157-35da-4cf7-a755-86123f5e09a0/volumes" Oct 11 10:57:00.647619 master-1 kubenswrapper[4771]: I1011 10:57:00.647497 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-scripts\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:57:00.653675 master-1 kubenswrapper[4771]: I1011 10:57:00.653608 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-scripts\") pod \"ceilometer-0\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " pod="openstack/ceilometer-0" Oct 11 10:57:00.672456 master-1 kubenswrapper[4771]: I1011 10:57:00.672390 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:57:00.899404 master-1 kubenswrapper[4771]: E1011 10:57:00.898832 4771 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/f67f5f4f9f37aac275f10afbe09a8c9fa57c1952c80ddcea69a513171faa9df7/diff" to get inode usage: stat /var/lib/containers/storage/overlay/f67f5f4f9f37aac275f10afbe09a8c9fa57c1952c80ddcea69a513171faa9df7/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openstack_ceilometer-0_d0cc5394-b33f-41a9-bbe2-d772e75a8f58/ceilometer-central-agent/0.log" to get inode usage: stat /var/log/pods/openstack_ceilometer-0_d0cc5394-b33f-41a9-bbe2-d772e75a8f58/ceilometer-central-agent/0.log: no such file or directory Oct 11 10:57:01.158670 master-1 kubenswrapper[4771]: I1011 10:57:01.158624 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:01.176577 master-1 kubenswrapper[4771]: I1011 10:57:01.176545 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 10:57:01.631067 master-1 kubenswrapper[4771]: I1011 10:57:01.630994 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerStarted","Data":"26bb9eb9589e069594c3ca946945eba89d16e626b603cd344d3ddae2ac0f7ded"} Oct 11 10:57:01.807742 master-1 kubenswrapper[4771]: E1011 10:57:01.807673 4771 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/55955291c1ae6b17a0bb7ff47fc06e286ff1749fa260fead2a93c54d085cacc9/diff" to get inode usage: stat /var/lib/containers/storage/overlay/55955291c1ae6b17a0bb7ff47fc06e286ff1749fa260fead2a93c54d085cacc9/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openstack_ceilometer-0_d0cc5394-b33f-41a9-bbe2-d772e75a8f58/ceilometer-notification-agent/0.log" to get inode usage: stat /var/log/pods/openstack_ceilometer-0_d0cc5394-b33f-41a9-bbe2-d772e75a8f58/ceilometer-notification-agent/0.log: no such file or directory Oct 11 10:57:01.942741 master-1 kubenswrapper[4771]: I1011 10:57:01.942635 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:01.943035 master-1 kubenswrapper[4771]: I1011 10:57:01.942895 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/nova-scheduler-1" podUID="4619bcb1-090e-4824-adfe-6a526158d0ea" containerName="nova-scheduler-scheduler" containerID="cri-o://a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" gracePeriod=30 Oct 11 10:57:02.488379 master-1 kubenswrapper[4771]: E1011 10:57:02.487702 4771 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/b6dfdc9be6dfc3040d4a1da475e8e872f701d8009c5039d372318d1f45772fb0/diff" to get inode usage: stat /var/lib/containers/storage/overlay/b6dfdc9be6dfc3040d4a1da475e8e872f701d8009c5039d372318d1f45772fb0/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openstack_ceilometer-0_d0cc5394-b33f-41a9-bbe2-d772e75a8f58/sg-core/0.log" to get inode usage: stat /var/log/pods/openstack_ceilometer-0_d0cc5394-b33f-41a9-bbe2-d772e75a8f58/sg-core/0.log: no such file or directory Oct 11 10:57:02.589768 master-1 kubenswrapper[4771]: E1011 10:57:02.589700 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" cmd=["/usr/bin/pgrep","-r","DRST","nova-scheduler"] Oct 11 10:57:02.591485 master-1 kubenswrapper[4771]: E1011 10:57:02.591390 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" cmd=["/usr/bin/pgrep","-r","DRST","nova-scheduler"] Oct 11 10:57:02.592772 master-1 kubenswrapper[4771]: E1011 10:57:02.592724 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" cmd=["/usr/bin/pgrep","-r","DRST","nova-scheduler"] Oct 11 10:57:02.592898 master-1 kubenswrapper[4771]: E1011 10:57:02.592771 4771 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openstack/nova-scheduler-1" podUID="4619bcb1-090e-4824-adfe-6a526158d0ea" containerName="nova-scheduler-scheduler" Oct 11 10:57:02.642581 master-1 kubenswrapper[4771]: I1011 10:57:02.642502 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerStarted","Data":"dd5ef923fe19c8499d59b13d67599b3bf99730453e82ff9eb110343bd5333a66"} Oct 11 10:57:02.642581 master-1 kubenswrapper[4771]: I1011 10:57:02.642560 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerStarted","Data":"329f7fd8063c9aaba98ccaa627d41506880615f2c94e9a537c2fbe8e686bc8be"} Oct 11 10:57:03.660016 master-1 kubenswrapper[4771]: I1011 10:57:03.659908 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerStarted","Data":"06b1b45211e1a6439784f737849060952f6c89a6f37d2a3b15906929efc29bff"} Oct 11 10:57:04.212430 master-1 kubenswrapper[4771]: E1011 10:57:04.212325 4771 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/ff4f35cc08d7b02affab5f4eeaf66f8103ae06a636fc32c1be940b7334c4f803/diff" to get inode usage: stat /var/lib/containers/storage/overlay/ff4f35cc08d7b02affab5f4eeaf66f8103ae06a636fc32c1be940b7334c4f803/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openstack_ceilometer-0_d0cc5394-b33f-41a9-bbe2-d772e75a8f58/proxy-httpd/0.log" to get inode usage: stat /var/log/pods/openstack_ceilometer-0_d0cc5394-b33f-41a9-bbe2-d772e75a8f58/proxy-httpd/0.log: no such file or directory Oct 11 10:57:05.379846 master-1 kubenswrapper[4771]: W1011 10:57:05.379765 4771 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9356157_35da_4cf7_a755_86123f5e09a0.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9356157_35da_4cf7_a755_86123f5e09a0.slice: no such file or directory Oct 11 10:57:05.394900 master-1 kubenswrapper[4771]: E1011 10:57:05.394758 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice/crio-4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-conmon-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:57:05.395199 master-1 kubenswrapper[4771]: E1011 10:57:05.394942 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-conmon-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:57:05.395313 master-1 kubenswrapper[4771]: E1011 10:57:05.394947 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-conmon-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice/crio-4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344\": RecentStats: unable to find data in memory cache]" Oct 11 10:57:05.395453 master-1 kubenswrapper[4771]: E1011 10:57:05.395183 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-conmon-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice/crio-4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85d5cfa_8073_4bbf_9eff_78fde719dadf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:57:05.395943 master-1 kubenswrapper[4771]: E1011 10:57:05.395767 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod602a8d3a_2ca2_43d2_8def_5718d9baf2ee.slice/crio-conmon-23ed8d2300382ac408577223ca4d96c6b722a1c94b3187394d9ec21991883547.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85d5cfa_8073_4bbf_9eff_78fde719dadf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice/crio-4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-conmon-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0\": RecentStats: unable to find data in memory cache]" Oct 11 10:57:05.397728 master-1 kubenswrapper[4771]: E1011 10:57:05.397649 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-conmon-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85d5cfa_8073_4bbf_9eff_78fde719dadf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85d5cfa_8073_4bbf_9eff_78fde719dadf.slice/crio-792c6e93eebb0025120939bb299c7a87876ec3dbbd22c047c0d886532fa269ba\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:57:05.399887 master-1 kubenswrapper[4771]: E1011 10:57:05.399795 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-conmon-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85d5cfa_8073_4bbf_9eff_78fde719dadf.slice/crio-792c6e93eebb0025120939bb299c7a87876ec3dbbd22c047c0d886532fa269ba\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice/crio-4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85d5cfa_8073_4bbf_9eff_78fde719dadf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache]" Oct 11 10:57:05.401956 master-1 kubenswrapper[4771]: E1011 10:57:05.401870 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-conmon-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85d5cfa_8073_4bbf_9eff_78fde719dadf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-conmon-2c5f1db8917c8af20f15f8f5c86b116c03c3cf84afbea6d406851b9dc2d31536.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-6adea3e5f64531e0c86b74d225c5177d037bfd577bffef9095ea2e44e640d111.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice/crio-4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-505e62311b6ed5c9ff0953d8d85544302f51dd00d4e8b39726a1f15af9f39dfe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-dfcca484b1ef8ab38e804d6ef394f26fc64c3bf3d7f1246b3ca103ffc5677a68.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-3a34a73cdd7c1cec4079fa29156740bcbf8771fe95c00057f168b2498eac713d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice\": RecentStats: unable to find data in memory cache]" Oct 11 10:57:05.405319 master-1 kubenswrapper[4771]: E1011 10:57:05.405199 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b5e37e3_9afd_4ff3_b992_1e6c28a986ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b5e37e3_9afd_4ff3_b992_1e6c28a986ad.slice/crio-8225c71dadd5dd5d7cdb7b603f12129b97565dbfb98b6f8553a5f73b645e62cc\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe7833d_9251_4545_ba68_f58c146188f1.slice/crio-fa2bc31890cd28c5ae042f31a82dc462c8d6a57c6fbf10c4706aaa08a519f43e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85d5cfa_8073_4bbf_9eff_78fde719dadf.slice/crio-792c6e93eebb0025120939bb299c7a87876ec3dbbd22c047c0d886532fa269ba\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice/crio-4b4427ca1aac1d5de2fce6df6d3c919384d14bf07e538280903548b74f73e344\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice/crio-c62163629aca98833fc3692d5fe6b9a44972b443acaee8cb73af5daad3f74fd0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod602a8d3a_2ca2_43d2_8def_5718d9baf2ee.slice/crio-conmon-23ed8d2300382ac408577223ca4d96c6b722a1c94b3187394d9ec21991883547.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cc5394_b33f_41a9_bbe2_d772e75a8f58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85d5cfa_8073_4bbf_9eff_78fde719dadf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de492fb_5249_49e2_a327_756234aa92bd.slice\": RecentStats: unable to find data in memory cache]" Oct 11 10:57:05.694805 master-1 kubenswrapper[4771]: I1011 10:57:05.694683 4771 generic.go:334] "Generic (PLEG): container finished" podID="602a8d3a-2ca2-43d2-8def-5718d9baf2ee" containerID="23ed8d2300382ac408577223ca4d96c6b722a1c94b3187394d9ec21991883547" exitCode=137 Oct 11 10:57:05.694805 master-1 kubenswrapper[4771]: I1011 10:57:05.694767 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-novncproxy-0" event={"ID":"602a8d3a-2ca2-43d2-8def-5718d9baf2ee","Type":"ContainerDied","Data":"23ed8d2300382ac408577223ca4d96c6b722a1c94b3187394d9ec21991883547"} Oct 11 10:57:05.700305 master-1 kubenswrapper[4771]: I1011 10:57:05.696582 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerStarted","Data":"1856a76e1ae1c62aaea063758c5bd874b8a3b66130c9fe8eb095d5cef903001f"} Oct 11 10:57:05.700305 master-1 kubenswrapper[4771]: I1011 10:57:05.696872 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 10:57:05.856834 master-1 kubenswrapper[4771]: I1011 10:57:05.856781 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:05.883408 master-1 kubenswrapper[4771]: I1011 10:57:05.883316 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=4.420750834 podStartE2EDuration="7.883296042s" podCreationTimestamp="2025-10-11 10:56:58 +0000 UTC" firstStartedPulling="2025-10-11 10:57:01.176497738 +0000 UTC m=+1853.150724189" lastFinishedPulling="2025-10-11 10:57:04.639042946 +0000 UTC m=+1856.613269397" observedRunningTime="2025-10-11 10:57:05.758183273 +0000 UTC m=+1857.732409714" watchObservedRunningTime="2025-10-11 10:57:05.883296042 +0000 UTC m=+1857.857522483" Oct 11 10:57:05.889435 master-1 kubenswrapper[4771]: I1011 10:57:05.889391 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-config-data\") pod \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " Oct 11 10:57:05.889608 master-1 kubenswrapper[4771]: I1011 10:57:05.889455 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-combined-ca-bundle\") pod \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " Oct 11 10:57:05.889760 master-1 kubenswrapper[4771]: I1011 10:57:05.889730 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5zqd\" (UniqueName: \"kubernetes.io/projected/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-kube-api-access-g5zqd\") pod \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\" (UID: \"602a8d3a-2ca2-43d2-8def-5718d9baf2ee\") " Oct 11 10:57:05.901438 master-1 kubenswrapper[4771]: I1011 10:57:05.901348 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-kube-api-access-g5zqd" (OuterVolumeSpecName: "kube-api-access-g5zqd") pod "602a8d3a-2ca2-43d2-8def-5718d9baf2ee" (UID: "602a8d3a-2ca2-43d2-8def-5718d9baf2ee"). InnerVolumeSpecName "kube-api-access-g5zqd". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:05.913396 master-1 kubenswrapper[4771]: I1011 10:57:05.913325 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-config-data" (OuterVolumeSpecName: "config-data") pod "602a8d3a-2ca2-43d2-8def-5718d9baf2ee" (UID: "602a8d3a-2ca2-43d2-8def-5718d9baf2ee"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:05.913535 master-1 kubenswrapper[4771]: I1011 10:57:05.913490 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "602a8d3a-2ca2-43d2-8def-5718d9baf2ee" (UID: "602a8d3a-2ca2-43d2-8def-5718d9baf2ee"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:05.991196 master-1 kubenswrapper[4771]: I1011 10:57:05.991134 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-g5zqd\" (UniqueName: \"kubernetes.io/projected/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-kube-api-access-g5zqd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:05.991196 master-1 kubenswrapper[4771]: I1011 10:57:05.991178 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:05.991196 master-1 kubenswrapper[4771]: I1011 10:57:05.991190 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/602a8d3a-2ca2-43d2-8def-5718d9baf2ee-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:06.720497 master-1 kubenswrapper[4771]: I1011 10:57:06.720438 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-novncproxy-0" event={"ID":"602a8d3a-2ca2-43d2-8def-5718d9baf2ee","Type":"ContainerDied","Data":"9b7f7d7c0af1b640b51f7a6b8a5687c5423669e8ea192915f2e34e079daaef17"} Oct 11 10:57:06.721004 master-1 kubenswrapper[4771]: I1011 10:57:06.720508 4771 scope.go:117] "RemoveContainer" containerID="23ed8d2300382ac408577223ca4d96c6b722a1c94b3187394d9ec21991883547" Oct 11 10:57:06.721004 master-1 kubenswrapper[4771]: I1011 10:57:06.720693 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.750206 master-1 kubenswrapper[4771]: I1011 10:57:06.750108 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-cell1-novncproxy-0"] Oct 11 10:57:06.760732 master-1 kubenswrapper[4771]: I1011 10:57:06.759281 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-cell1-novncproxy-0"] Oct 11 10:57:06.798249 master-1 kubenswrapper[4771]: I1011 10:57:06.798147 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell1-novncproxy-0"] Oct 11 10:57:06.798640 master-1 kubenswrapper[4771]: E1011 10:57:06.798617 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="602a8d3a-2ca2-43d2-8def-5718d9baf2ee" containerName="nova-cell1-novncproxy-novncproxy" Oct 11 10:57:06.798687 master-1 kubenswrapper[4771]: I1011 10:57:06.798639 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="602a8d3a-2ca2-43d2-8def-5718d9baf2ee" containerName="nova-cell1-novncproxy-novncproxy" Oct 11 10:57:06.798998 master-1 kubenswrapper[4771]: I1011 10:57:06.798842 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="602a8d3a-2ca2-43d2-8def-5718d9baf2ee" containerName="nova-cell1-novncproxy-novncproxy" Oct 11 10:57:06.799732 master-1 kubenswrapper[4771]: I1011 10:57:06.799706 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.802712 master-1 kubenswrapper[4771]: I1011 10:57:06.802674 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-nova-novncproxy-cell1-public-svc" Oct 11 10:57:06.803575 master-1 kubenswrapper[4771]: I1011 10:57:06.802970 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-nova-novncproxy-cell1-vencrypt" Oct 11 10:57:06.803575 master-1 kubenswrapper[4771]: I1011 10:57:06.803107 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell1-novncproxy-config-data" Oct 11 10:57:06.806537 master-1 kubenswrapper[4771]: I1011 10:57:06.806403 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45l2x\" (UniqueName: \"kubernetes.io/projected/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-kube-api-access-45l2x\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.806537 master-1 kubenswrapper[4771]: I1011 10:57:06.806474 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-novncproxy-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-nova-novncproxy-tls-certs\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.806537 master-1 kubenswrapper[4771]: I1011 10:57:06.806542 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-config-data\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.806686 master-1 kubenswrapper[4771]: I1011 10:57:06.806565 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"vencrypt-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-vencrypt-tls-certs\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.806686 master-1 kubenswrapper[4771]: I1011 10:57:06.806587 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-combined-ca-bundle\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.816719 master-1 kubenswrapper[4771]: I1011 10:57:06.816639 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-novncproxy-0"] Oct 11 10:57:06.908803 master-1 kubenswrapper[4771]: I1011 10:57:06.908719 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-45l2x\" (UniqueName: \"kubernetes.io/projected/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-kube-api-access-45l2x\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.909227 master-1 kubenswrapper[4771]: I1011 10:57:06.908841 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-novncproxy-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-nova-novncproxy-tls-certs\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.909227 master-1 kubenswrapper[4771]: I1011 10:57:06.908956 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-config-data\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.909227 master-1 kubenswrapper[4771]: I1011 10:57:06.908984 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"vencrypt-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-vencrypt-tls-certs\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.909227 master-1 kubenswrapper[4771]: I1011 10:57:06.909011 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-combined-ca-bundle\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.912855 master-1 kubenswrapper[4771]: I1011 10:57:06.912801 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"vencrypt-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-vencrypt-tls-certs\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.915203 master-1 kubenswrapper[4771]: I1011 10:57:06.915162 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-novncproxy-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-nova-novncproxy-tls-certs\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.915898 master-1 kubenswrapper[4771]: I1011 10:57:06.915855 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-combined-ca-bundle\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.915965 master-1 kubenswrapper[4771]: I1011 10:57:06.915895 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-config-data\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:06.935055 master-1 kubenswrapper[4771]: I1011 10:57:06.934986 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-45l2x\" (UniqueName: \"kubernetes.io/projected/1c2ae850-015d-40bc-8af0-b47b9bb6f46b-kube-api-access-45l2x\") pod \"nova-cell1-novncproxy-0\" (UID: \"1c2ae850-015d-40bc-8af0-b47b9bb6f46b\") " pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:07.115691 master-1 kubenswrapper[4771]: I1011 10:57:07.115599 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:07.575593 master-1 kubenswrapper[4771]: I1011 10:57:07.575536 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-novncproxy-0"] Oct 11 10:57:07.581918 master-1 kubenswrapper[4771]: W1011 10:57:07.580759 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2ae850_015d_40bc_8af0_b47b9bb6f46b.slice/crio-31ee81c7ef7c26bafc8de05589408cca2a0aa069e0c298182198521073763c3f WatchSource:0}: Error finding container 31ee81c7ef7c26bafc8de05589408cca2a0aa069e0c298182198521073763c3f: Status 404 returned error can't find the container with id 31ee81c7ef7c26bafc8de05589408cca2a0aa069e0c298182198521073763c3f Oct 11 10:57:07.581918 master-1 kubenswrapper[4771]: E1011 10:57:07.581515 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8 is running failed: container process not found" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" cmd=["/usr/bin/pgrep","-r","DRST","nova-scheduler"] Oct 11 10:57:07.581918 master-1 kubenswrapper[4771]: E1011 10:57:07.581834 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8 is running failed: container process not found" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" cmd=["/usr/bin/pgrep","-r","DRST","nova-scheduler"] Oct 11 10:57:07.582756 master-1 kubenswrapper[4771]: E1011 10:57:07.582348 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8 is running failed: container process not found" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" cmd=["/usr/bin/pgrep","-r","DRST","nova-scheduler"] Oct 11 10:57:07.582756 master-1 kubenswrapper[4771]: E1011 10:57:07.582425 4771 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8 is running failed: container process not found" probeType="Readiness" pod="openstack/nova-scheduler-1" podUID="4619bcb1-090e-4824-adfe-6a526158d0ea" containerName="nova-scheduler-scheduler" Oct 11 10:57:07.623164 master-1 kubenswrapper[4771]: I1011 10:57:07.623127 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:57:07.730091 master-1 kubenswrapper[4771]: I1011 10:57:07.730020 4771 generic.go:334] "Generic (PLEG): container finished" podID="4619bcb1-090e-4824-adfe-6a526158d0ea" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" exitCode=0 Oct 11 10:57:07.743729 master-1 kubenswrapper[4771]: I1011 10:57:07.730106 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:57:07.743729 master-1 kubenswrapper[4771]: I1011 10:57:07.730131 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"4619bcb1-090e-4824-adfe-6a526158d0ea","Type":"ContainerDied","Data":"a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8"} Oct 11 10:57:07.743729 master-1 kubenswrapper[4771]: I1011 10:57:07.730162 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"4619bcb1-090e-4824-adfe-6a526158d0ea","Type":"ContainerDied","Data":"2855d39e0600653f4ca98e1b1c4a631cd2cea811da489ab4ee595433738a99d4"} Oct 11 10:57:07.743729 master-1 kubenswrapper[4771]: I1011 10:57:07.730181 4771 scope.go:117] "RemoveContainer" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" Oct 11 10:57:07.743729 master-1 kubenswrapper[4771]: I1011 10:57:07.733310 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-novncproxy-0" event={"ID":"1c2ae850-015d-40bc-8af0-b47b9bb6f46b","Type":"ContainerStarted","Data":"31ee81c7ef7c26bafc8de05589408cca2a0aa069e0c298182198521073763c3f"} Oct 11 10:57:07.743729 master-1 kubenswrapper[4771]: I1011 10:57:07.739151 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-combined-ca-bundle\") pod \"4619bcb1-090e-4824-adfe-6a526158d0ea\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " Oct 11 10:57:07.743729 master-1 kubenswrapper[4771]: I1011 10:57:07.739298 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-config-data\") pod \"4619bcb1-090e-4824-adfe-6a526158d0ea\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " Oct 11 10:57:07.743729 master-1 kubenswrapper[4771]: I1011 10:57:07.739393 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srjhk\" (UniqueName: \"kubernetes.io/projected/4619bcb1-090e-4824-adfe-6a526158d0ea-kube-api-access-srjhk\") pod \"4619bcb1-090e-4824-adfe-6a526158d0ea\" (UID: \"4619bcb1-090e-4824-adfe-6a526158d0ea\") " Oct 11 10:57:07.743729 master-1 kubenswrapper[4771]: I1011 10:57:07.742229 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4619bcb1-090e-4824-adfe-6a526158d0ea-kube-api-access-srjhk" (OuterVolumeSpecName: "kube-api-access-srjhk") pod "4619bcb1-090e-4824-adfe-6a526158d0ea" (UID: "4619bcb1-090e-4824-adfe-6a526158d0ea"). InnerVolumeSpecName "kube-api-access-srjhk". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:07.752228 master-1 kubenswrapper[4771]: I1011 10:57:07.752187 4771 scope.go:117] "RemoveContainer" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" Oct 11 10:57:07.752698 master-1 kubenswrapper[4771]: E1011 10:57:07.752665 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8\": container with ID starting with a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8 not found: ID does not exist" containerID="a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8" Oct 11 10:57:07.752748 master-1 kubenswrapper[4771]: I1011 10:57:07.752701 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8"} err="failed to get container status \"a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8\": rpc error: code = NotFound desc = could not find container \"a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8\": container with ID starting with a23248db23cc07663daa71fc11e05668a4c12da23548369d411cb7d7393beab8 not found: ID does not exist" Oct 11 10:57:07.764892 master-1 kubenswrapper[4771]: I1011 10:57:07.764829 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-config-data" (OuterVolumeSpecName: "config-data") pod "4619bcb1-090e-4824-adfe-6a526158d0ea" (UID: "4619bcb1-090e-4824-adfe-6a526158d0ea"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:07.767738 master-1 kubenswrapper[4771]: I1011 10:57:07.767689 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "4619bcb1-090e-4824-adfe-6a526158d0ea" (UID: "4619bcb1-090e-4824-adfe-6a526158d0ea"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:07.841509 master-1 kubenswrapper[4771]: I1011 10:57:07.841430 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-srjhk\" (UniqueName: \"kubernetes.io/projected/4619bcb1-090e-4824-adfe-6a526158d0ea-kube-api-access-srjhk\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:07.841509 master-1 kubenswrapper[4771]: I1011 10:57:07.841478 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:07.841509 master-1 kubenswrapper[4771]: I1011 10:57:07.841489 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4619bcb1-090e-4824-adfe-6a526158d0ea-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:08.068957 master-1 kubenswrapper[4771]: I1011 10:57:08.068877 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:08.074267 master-1 kubenswrapper[4771]: I1011 10:57:08.074214 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:08.106060 master-1 kubenswrapper[4771]: I1011 10:57:08.105990 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:08.106470 master-1 kubenswrapper[4771]: E1011 10:57:08.106441 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="4619bcb1-090e-4824-adfe-6a526158d0ea" containerName="nova-scheduler-scheduler" Oct 11 10:57:08.106470 master-1 kubenswrapper[4771]: I1011 10:57:08.106464 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="4619bcb1-090e-4824-adfe-6a526158d0ea" containerName="nova-scheduler-scheduler" Oct 11 10:57:08.106683 master-1 kubenswrapper[4771]: I1011 10:57:08.106648 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="4619bcb1-090e-4824-adfe-6a526158d0ea" containerName="nova-scheduler-scheduler" Oct 11 10:57:08.107596 master-1 kubenswrapper[4771]: I1011 10:57:08.107566 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:57:08.110454 master-1 kubenswrapper[4771]: I1011 10:57:08.110406 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-scheduler-config-data" Oct 11 10:57:08.129974 master-1 kubenswrapper[4771]: I1011 10:57:08.129907 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:08.247442 master-1 kubenswrapper[4771]: I1011 10:57:08.247212 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-combined-ca-bundle\") pod \"nova-scheduler-1\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:08.247442 master-1 kubenswrapper[4771]: I1011 10:57:08.247275 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5m8\" (UniqueName: \"kubernetes.io/projected/82101e85-023a-4398-bb5e-4162dea69f46-kube-api-access-kh5m8\") pod \"nova-scheduler-1\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:08.247683 master-1 kubenswrapper[4771]: I1011 10:57:08.247534 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-config-data\") pod \"nova-scheduler-1\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:08.350374 master-1 kubenswrapper[4771]: I1011 10:57:08.350270 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-config-data\") pod \"nova-scheduler-1\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:08.350635 master-1 kubenswrapper[4771]: I1011 10:57:08.350533 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-combined-ca-bundle\") pod \"nova-scheduler-1\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:08.350635 master-1 kubenswrapper[4771]: I1011 10:57:08.350581 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5m8\" (UniqueName: \"kubernetes.io/projected/82101e85-023a-4398-bb5e-4162dea69f46-kube-api-access-kh5m8\") pod \"nova-scheduler-1\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:08.376177 master-1 kubenswrapper[4771]: I1011 10:57:08.376115 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-combined-ca-bundle\") pod \"nova-scheduler-1\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:08.376470 master-1 kubenswrapper[4771]: I1011 10:57:08.376426 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-config-data\") pod \"nova-scheduler-1\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:08.382252 master-1 kubenswrapper[4771]: I1011 10:57:08.381703 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5m8\" (UniqueName: \"kubernetes.io/projected/82101e85-023a-4398-bb5e-4162dea69f46-kube-api-access-kh5m8\") pod \"nova-scheduler-1\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:08.435158 master-1 kubenswrapper[4771]: I1011 10:57:08.435053 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:57:08.454316 master-1 kubenswrapper[4771]: I1011 10:57:08.454254 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4619bcb1-090e-4824-adfe-6a526158d0ea" path="/var/lib/kubelet/pods/4619bcb1-090e-4824-adfe-6a526158d0ea/volumes" Oct 11 10:57:08.459223 master-1 kubenswrapper[4771]: I1011 10:57:08.459161 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="602a8d3a-2ca2-43d2-8def-5718d9baf2ee" path="/var/lib/kubelet/pods/602a8d3a-2ca2-43d2-8def-5718d9baf2ee/volumes" Oct 11 10:57:08.745724 master-1 kubenswrapper[4771]: I1011 10:57:08.745619 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-novncproxy-0" event={"ID":"1c2ae850-015d-40bc-8af0-b47b9bb6f46b","Type":"ContainerStarted","Data":"6a8f41b531ca846c95fe153a07f06cfef2fda05873909ec82bf0978ab3366378"} Oct 11 10:57:08.783938 master-1 kubenswrapper[4771]: I1011 10:57:08.779648 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell1-novncproxy-0" podStartSLOduration=2.779627337 podStartE2EDuration="2.779627337s" podCreationTimestamp="2025-10-11 10:57:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:57:08.772579761 +0000 UTC m=+1860.746806222" watchObservedRunningTime="2025-10-11 10:57:08.779627337 +0000 UTC m=+1860.753853788" Oct 11 10:57:08.990768 master-1 kubenswrapper[4771]: W1011 10:57:08.990690 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82101e85_023a_4398_bb5e_4162dea69f46.slice/crio-344c84a1c3165ebb54446b68471bee6244a2b1504eca4e8fa46ae99da6e9b301 WatchSource:0}: Error finding container 344c84a1c3165ebb54446b68471bee6244a2b1504eca4e8fa46ae99da6e9b301: Status 404 returned error can't find the container with id 344c84a1c3165ebb54446b68471bee6244a2b1504eca4e8fa46ae99da6e9b301 Oct 11 10:57:08.993372 master-1 kubenswrapper[4771]: I1011 10:57:08.992946 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:09.049379 master-1 kubenswrapper[4771]: I1011 10:57:09.049269 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:09.050160 master-1 kubenswrapper[4771]: I1011 10:57:09.050060 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/nova-metadata-1" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-metadata" containerID="cri-o://cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae" gracePeriod=30 Oct 11 10:57:09.071550 master-1 kubenswrapper[4771]: I1011 10:57:09.063724 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/nova-metadata-1" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-log" containerID="cri-o://10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f" gracePeriod=30 Oct 11 10:57:09.762506 master-1 kubenswrapper[4771]: I1011 10:57:09.762433 4771 generic.go:334] "Generic (PLEG): container finished" podID="1596746b-25ca-487a-9e49-93e532f2838b" containerID="10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f" exitCode=143 Oct 11 10:57:09.763424 master-1 kubenswrapper[4771]: I1011 10:57:09.762530 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"1596746b-25ca-487a-9e49-93e532f2838b","Type":"ContainerDied","Data":"10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f"} Oct 11 10:57:09.764531 master-1 kubenswrapper[4771]: I1011 10:57:09.764486 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"82101e85-023a-4398-bb5e-4162dea69f46","Type":"ContainerStarted","Data":"ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc"} Oct 11 10:57:09.764698 master-1 kubenswrapper[4771]: I1011 10:57:09.764600 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"82101e85-023a-4398-bb5e-4162dea69f46","Type":"ContainerStarted","Data":"344c84a1c3165ebb54446b68471bee6244a2b1504eca4e8fa46ae99da6e9b301"} Oct 11 10:57:09.811799 master-1 kubenswrapper[4771]: I1011 10:57:09.811716 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-scheduler-1" podStartSLOduration=1.811689355 podStartE2EDuration="1.811689355s" podCreationTimestamp="2025-10-11 10:57:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:57:09.793460363 +0000 UTC m=+1861.767686864" watchObservedRunningTime="2025-10-11 10:57:09.811689355 +0000 UTC m=+1861.785915796" Oct 11 10:57:12.117315 master-1 kubenswrapper[4771]: I1011 10:57:12.117224 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:12.118146 master-1 kubenswrapper[4771]: I1011 10:57:12.118093 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:12.118288 master-1 kubenswrapper[4771]: I1011 10:57:12.118218 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="ceilometer-central-agent" containerID="cri-o://329f7fd8063c9aaba98ccaa627d41506880615f2c94e9a537c2fbe8e686bc8be" gracePeriod=30 Oct 11 10:57:12.118412 master-1 kubenswrapper[4771]: I1011 10:57:12.118385 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="ceilometer-notification-agent" containerID="cri-o://dd5ef923fe19c8499d59b13d67599b3bf99730453e82ff9eb110343bd5333a66" gracePeriod=30 Oct 11 10:57:12.118480 master-1 kubenswrapper[4771]: I1011 10:57:12.118433 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="sg-core" containerID="cri-o://06b1b45211e1a6439784f737849060952f6c89a6f37d2a3b15906929efc29bff" gracePeriod=30 Oct 11 10:57:12.118528 master-1 kubenswrapper[4771]: I1011 10:57:12.118444 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="proxy-httpd" containerID="cri-o://1856a76e1ae1c62aaea063758c5bd874b8a3b66130c9fe8eb095d5cef903001f" gracePeriod=30 Oct 11 10:57:12.764409 master-1 kubenswrapper[4771]: I1011 10:57:12.764019 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:57:12.812678 master-1 kubenswrapper[4771]: I1011 10:57:12.812502 4771 generic.go:334] "Generic (PLEG): container finished" podID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerID="1856a76e1ae1c62aaea063758c5bd874b8a3b66130c9fe8eb095d5cef903001f" exitCode=0 Oct 11 10:57:12.812678 master-1 kubenswrapper[4771]: I1011 10:57:12.812549 4771 generic.go:334] "Generic (PLEG): container finished" podID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerID="06b1b45211e1a6439784f737849060952f6c89a6f37d2a3b15906929efc29bff" exitCode=2 Oct 11 10:57:12.812678 master-1 kubenswrapper[4771]: I1011 10:57:12.812559 4771 generic.go:334] "Generic (PLEG): container finished" podID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerID="dd5ef923fe19c8499d59b13d67599b3bf99730453e82ff9eb110343bd5333a66" exitCode=0 Oct 11 10:57:12.812678 master-1 kubenswrapper[4771]: I1011 10:57:12.812568 4771 generic.go:334] "Generic (PLEG): container finished" podID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerID="329f7fd8063c9aaba98ccaa627d41506880615f2c94e9a537c2fbe8e686bc8be" exitCode=0 Oct 11 10:57:12.812678 master-1 kubenswrapper[4771]: I1011 10:57:12.812566 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerDied","Data":"1856a76e1ae1c62aaea063758c5bd874b8a3b66130c9fe8eb095d5cef903001f"} Oct 11 10:57:12.812678 master-1 kubenswrapper[4771]: I1011 10:57:12.812639 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerDied","Data":"06b1b45211e1a6439784f737849060952f6c89a6f37d2a3b15906929efc29bff"} Oct 11 10:57:12.812678 master-1 kubenswrapper[4771]: I1011 10:57:12.812656 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerDied","Data":"dd5ef923fe19c8499d59b13d67599b3bf99730453e82ff9eb110343bd5333a66"} Oct 11 10:57:12.812678 master-1 kubenswrapper[4771]: I1011 10:57:12.812669 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerDied","Data":"329f7fd8063c9aaba98ccaa627d41506880615f2c94e9a537c2fbe8e686bc8be"} Oct 11 10:57:12.817951 master-1 kubenswrapper[4771]: I1011 10:57:12.817803 4771 generic.go:334] "Generic (PLEG): container finished" podID="1596746b-25ca-487a-9e49-93e532f2838b" containerID="cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae" exitCode=0 Oct 11 10:57:12.817951 master-1 kubenswrapper[4771]: I1011 10:57:12.817863 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"1596746b-25ca-487a-9e49-93e532f2838b","Type":"ContainerDied","Data":"cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae"} Oct 11 10:57:12.817951 master-1 kubenswrapper[4771]: I1011 10:57:12.817889 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:57:12.817951 master-1 kubenswrapper[4771]: I1011 10:57:12.817907 4771 scope.go:117] "RemoveContainer" containerID="cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae" Oct 11 10:57:12.818184 master-1 kubenswrapper[4771]: I1011 10:57:12.817891 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"1596746b-25ca-487a-9e49-93e532f2838b","Type":"ContainerDied","Data":"089dbb4c74bf7c97de17802c964262c34b58cf3f278035cc8b343f4df54f1f61"} Oct 11 10:57:12.872556 master-1 kubenswrapper[4771]: I1011 10:57:12.872507 4771 scope.go:117] "RemoveContainer" containerID="10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f" Oct 11 10:57:12.885130 master-1 kubenswrapper[4771]: I1011 10:57:12.885064 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-config-data\") pod \"1596746b-25ca-487a-9e49-93e532f2838b\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " Oct 11 10:57:12.885252 master-1 kubenswrapper[4771]: I1011 10:57:12.885209 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-combined-ca-bundle\") pod \"1596746b-25ca-487a-9e49-93e532f2838b\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " Oct 11 10:57:12.885317 master-1 kubenswrapper[4771]: I1011 10:57:12.885252 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scqmr\" (UniqueName: \"kubernetes.io/projected/1596746b-25ca-487a-9e49-93e532f2838b-kube-api-access-scqmr\") pod \"1596746b-25ca-487a-9e49-93e532f2838b\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " Oct 11 10:57:12.885495 master-1 kubenswrapper[4771]: I1011 10:57:12.885444 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/1596746b-25ca-487a-9e49-93e532f2838b-logs\") pod \"1596746b-25ca-487a-9e49-93e532f2838b\" (UID: \"1596746b-25ca-487a-9e49-93e532f2838b\") " Oct 11 10:57:12.886713 master-1 kubenswrapper[4771]: I1011 10:57:12.886669 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1596746b-25ca-487a-9e49-93e532f2838b-logs" (OuterVolumeSpecName: "logs") pod "1596746b-25ca-487a-9e49-93e532f2838b" (UID: "1596746b-25ca-487a-9e49-93e532f2838b"). InnerVolumeSpecName "logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:57:12.889880 master-1 kubenswrapper[4771]: I1011 10:57:12.889508 4771 scope.go:117] "RemoveContainer" containerID="cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae" Oct 11 10:57:12.890621 master-1 kubenswrapper[4771]: I1011 10:57:12.889981 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1596746b-25ca-487a-9e49-93e532f2838b-kube-api-access-scqmr" (OuterVolumeSpecName: "kube-api-access-scqmr") pod "1596746b-25ca-487a-9e49-93e532f2838b" (UID: "1596746b-25ca-487a-9e49-93e532f2838b"). InnerVolumeSpecName "kube-api-access-scqmr". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:12.890621 master-1 kubenswrapper[4771]: E1011 10:57:12.889988 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae\": container with ID starting with cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae not found: ID does not exist" containerID="cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae" Oct 11 10:57:12.890621 master-1 kubenswrapper[4771]: I1011 10:57:12.890071 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae"} err="failed to get container status \"cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae\": rpc error: code = NotFound desc = could not find container \"cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae\": container with ID starting with cc430b45b80be084cd02e8ad7b476be42a8b108197f68c496397cf28d0b094ae not found: ID does not exist" Oct 11 10:57:12.890621 master-1 kubenswrapper[4771]: I1011 10:57:12.890113 4771 scope.go:117] "RemoveContainer" containerID="10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f" Oct 11 10:57:12.890855 master-1 kubenswrapper[4771]: E1011 10:57:12.890626 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f\": container with ID starting with 10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f not found: ID does not exist" containerID="10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f" Oct 11 10:57:12.890855 master-1 kubenswrapper[4771]: I1011 10:57:12.890676 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f"} err="failed to get container status \"10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f\": rpc error: code = NotFound desc = could not find container \"10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f\": container with ID starting with 10394d9815d51c3e2ede873c2bea5e3b152ea84d8b7c0cf8df31f7c51efb349f not found: ID does not exist" Oct 11 10:57:12.919413 master-1 kubenswrapper[4771]: I1011 10:57:12.914739 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "1596746b-25ca-487a-9e49-93e532f2838b" (UID: "1596746b-25ca-487a-9e49-93e532f2838b"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:12.926814 master-1 kubenswrapper[4771]: I1011 10:57:12.926766 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-config-data" (OuterVolumeSpecName: "config-data") pod "1596746b-25ca-487a-9e49-93e532f2838b" (UID: "1596746b-25ca-487a-9e49-93e532f2838b"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:12.927235 master-1 kubenswrapper[4771]: I1011 10:57:12.927200 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:57:12.987459 master-1 kubenswrapper[4771]: I1011 10:57:12.987392 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-scripts\") pod \"9a279d25-518e-4a12-8b75-e3781fb22f05\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " Oct 11 10:57:12.987667 master-1 kubenswrapper[4771]: I1011 10:57:12.987506 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-combined-ca-bundle\") pod \"9a279d25-518e-4a12-8b75-e3781fb22f05\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " Oct 11 10:57:12.987667 master-1 kubenswrapper[4771]: I1011 10:57:12.987555 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-run-httpd\") pod \"9a279d25-518e-4a12-8b75-e3781fb22f05\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " Oct 11 10:57:12.987667 master-1 kubenswrapper[4771]: I1011 10:57:12.987619 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtfvl\" (UniqueName: \"kubernetes.io/projected/9a279d25-518e-4a12-8b75-e3781fb22f05-kube-api-access-wtfvl\") pod \"9a279d25-518e-4a12-8b75-e3781fb22f05\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " Oct 11 10:57:12.987667 master-1 kubenswrapper[4771]: I1011 10:57:12.987662 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-config-data\") pod \"9a279d25-518e-4a12-8b75-e3781fb22f05\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " Oct 11 10:57:12.987887 master-1 kubenswrapper[4771]: I1011 10:57:12.987696 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-log-httpd\") pod \"9a279d25-518e-4a12-8b75-e3781fb22f05\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " Oct 11 10:57:12.987887 master-1 kubenswrapper[4771]: I1011 10:57:12.987720 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-ceilometer-tls-certs\") pod \"9a279d25-518e-4a12-8b75-e3781fb22f05\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " Oct 11 10:57:12.987887 master-1 kubenswrapper[4771]: I1011 10:57:12.987760 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-sg-core-conf-yaml\") pod \"9a279d25-518e-4a12-8b75-e3781fb22f05\" (UID: \"9a279d25-518e-4a12-8b75-e3781fb22f05\") " Oct 11 10:57:12.988005 master-1 kubenswrapper[4771]: I1011 10:57:12.987969 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "9a279d25-518e-4a12-8b75-e3781fb22f05" (UID: "9a279d25-518e-4a12-8b75-e3781fb22f05"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:57:12.988226 master-1 kubenswrapper[4771]: I1011 10:57:12.988099 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "9a279d25-518e-4a12-8b75-e3781fb22f05" (UID: "9a279d25-518e-4a12-8b75-e3781fb22f05"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:57:12.990158 master-1 kubenswrapper[4771]: I1011 10:57:12.990117 4771 reconciler_common.go:293] "Volume detached for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/1596746b-25ca-487a-9e49-93e532f2838b-logs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:12.990158 master-1 kubenswrapper[4771]: I1011 10:57:12.990157 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:12.990304 master-1 kubenswrapper[4771]: I1011 10:57:12.990176 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1596746b-25ca-487a-9e49-93e532f2838b-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:12.990304 master-1 kubenswrapper[4771]: I1011 10:57:12.990198 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-scqmr\" (UniqueName: \"kubernetes.io/projected/1596746b-25ca-487a-9e49-93e532f2838b-kube-api-access-scqmr\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:12.990304 master-1 kubenswrapper[4771]: I1011 10:57:12.990222 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:12.990304 master-1 kubenswrapper[4771]: I1011 10:57:12.990235 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/9a279d25-518e-4a12-8b75-e3781fb22f05-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:13.003704 master-1 kubenswrapper[4771]: I1011 10:57:13.003632 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-scripts" (OuterVolumeSpecName: "scripts") pod "9a279d25-518e-4a12-8b75-e3781fb22f05" (UID: "9a279d25-518e-4a12-8b75-e3781fb22f05"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:13.007255 master-1 kubenswrapper[4771]: I1011 10:57:13.007192 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a279d25-518e-4a12-8b75-e3781fb22f05-kube-api-access-wtfvl" (OuterVolumeSpecName: "kube-api-access-wtfvl") pod "9a279d25-518e-4a12-8b75-e3781fb22f05" (UID: "9a279d25-518e-4a12-8b75-e3781fb22f05"). InnerVolumeSpecName "kube-api-access-wtfvl". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:13.037615 master-1 kubenswrapper[4771]: I1011 10:57:13.037545 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-ceilometer-tls-certs" (OuterVolumeSpecName: "ceilometer-tls-certs") pod "9a279d25-518e-4a12-8b75-e3781fb22f05" (UID: "9a279d25-518e-4a12-8b75-e3781fb22f05"). InnerVolumeSpecName "ceilometer-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:13.047171 master-1 kubenswrapper[4771]: I1011 10:57:13.047117 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "9a279d25-518e-4a12-8b75-e3781fb22f05" (UID: "9a279d25-518e-4a12-8b75-e3781fb22f05"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:13.048294 master-1 kubenswrapper[4771]: I1011 10:57:13.048245 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "9a279d25-518e-4a12-8b75-e3781fb22f05" (UID: "9a279d25-518e-4a12-8b75-e3781fb22f05"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:13.086714 master-1 kubenswrapper[4771]: I1011 10:57:13.085573 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-config-data" (OuterVolumeSpecName: "config-data") pod "9a279d25-518e-4a12-8b75-e3781fb22f05" (UID: "9a279d25-518e-4a12-8b75-e3781fb22f05"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:13.094490 master-1 kubenswrapper[4771]: I1011 10:57:13.093389 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-wtfvl\" (UniqueName: \"kubernetes.io/projected/9a279d25-518e-4a12-8b75-e3781fb22f05-kube-api-access-wtfvl\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:13.094490 master-1 kubenswrapper[4771]: I1011 10:57:13.093433 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:13.094490 master-1 kubenswrapper[4771]: I1011 10:57:13.093446 4771 reconciler_common.go:293] "Volume detached for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-ceilometer-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:13.094490 master-1 kubenswrapper[4771]: I1011 10:57:13.093457 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:13.094490 master-1 kubenswrapper[4771]: I1011 10:57:13.093466 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:13.094490 master-1 kubenswrapper[4771]: I1011 10:57:13.093475 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a279d25-518e-4a12-8b75-e3781fb22f05-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:13.158267 master-1 kubenswrapper[4771]: I1011 10:57:13.158195 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:13.173777 master-1 kubenswrapper[4771]: I1011 10:57:13.173635 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:13.196139 master-1 kubenswrapper[4771]: I1011 10:57:13.196076 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:13.196411 master-1 kubenswrapper[4771]: E1011 10:57:13.196390 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="ceilometer-central-agent" Oct 11 10:57:13.196411 master-1 kubenswrapper[4771]: I1011 10:57:13.196406 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="ceilometer-central-agent" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: E1011 10:57:13.196420 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="sg-core" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: I1011 10:57:13.196427 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="sg-core" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: E1011 10:57:13.196440 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-metadata" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: I1011 10:57:13.196449 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-metadata" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: E1011 10:57:13.196457 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="ceilometer-notification-agent" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: I1011 10:57:13.196463 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="ceilometer-notification-agent" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: E1011 10:57:13.196475 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="proxy-httpd" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: I1011 10:57:13.196481 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="proxy-httpd" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: E1011 10:57:13.196495 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-log" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: I1011 10:57:13.196501 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-log" Oct 11 10:57:13.196632 master-1 kubenswrapper[4771]: I1011 10:57:13.196630 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-metadata" Oct 11 10:57:13.197127 master-1 kubenswrapper[4771]: I1011 10:57:13.196650 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="ceilometer-central-agent" Oct 11 10:57:13.197127 master-1 kubenswrapper[4771]: I1011 10:57:13.196661 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="1596746b-25ca-487a-9e49-93e532f2838b" containerName="nova-metadata-log" Oct 11 10:57:13.197127 master-1 kubenswrapper[4771]: I1011 10:57:13.196674 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="ceilometer-notification-agent" Oct 11 10:57:13.197127 master-1 kubenswrapper[4771]: I1011 10:57:13.196686 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="sg-core" Oct 11 10:57:13.197127 master-1 kubenswrapper[4771]: I1011 10:57:13.196696 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" containerName="proxy-httpd" Oct 11 10:57:13.197640 master-1 kubenswrapper[4771]: I1011 10:57:13.197608 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:57:13.201159 master-1 kubenswrapper[4771]: I1011 10:57:13.201116 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-metadata-config-data" Oct 11 10:57:13.201874 master-1 kubenswrapper[4771]: I1011 10:57:13.201844 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-nova-metadata-internal-svc" Oct 11 10:57:13.246538 master-1 kubenswrapper[4771]: I1011 10:57:13.246470 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:13.305710 master-1 kubenswrapper[4771]: I1011 10:57:13.305666 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/486db0a3-f081-43d5-b20d-d7386531632e-logs\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.305801 master-1 kubenswrapper[4771]: I1011 10:57:13.305758 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-combined-ca-bundle\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.305943 master-1 kubenswrapper[4771]: I1011 10:57:13.305913 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-metadata-tls-certs\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-nova-metadata-tls-certs\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.306216 master-1 kubenswrapper[4771]: I1011 10:57:13.305998 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbq4h\" (UniqueName: \"kubernetes.io/projected/486db0a3-f081-43d5-b20d-d7386531632e-kube-api-access-lbq4h\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.306216 master-1 kubenswrapper[4771]: I1011 10:57:13.306048 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-config-data\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.407660 master-1 kubenswrapper[4771]: I1011 10:57:13.407581 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-lbq4h\" (UniqueName: \"kubernetes.io/projected/486db0a3-f081-43d5-b20d-d7386531632e-kube-api-access-lbq4h\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.407899 master-1 kubenswrapper[4771]: I1011 10:57:13.407675 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-config-data\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.407899 master-1 kubenswrapper[4771]: I1011 10:57:13.407782 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/486db0a3-f081-43d5-b20d-d7386531632e-logs\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.407899 master-1 kubenswrapper[4771]: I1011 10:57:13.407832 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-combined-ca-bundle\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.407899 master-1 kubenswrapper[4771]: I1011 10:57:13.407868 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-metadata-tls-certs\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-nova-metadata-tls-certs\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.409250 master-1 kubenswrapper[4771]: I1011 10:57:13.409187 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/486db0a3-f081-43d5-b20d-d7386531632e-logs\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.413319 master-1 kubenswrapper[4771]: I1011 10:57:13.412999 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-metadata-tls-certs\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-nova-metadata-tls-certs\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.414923 master-1 kubenswrapper[4771]: I1011 10:57:13.414870 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-config-data\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.415651 master-1 kubenswrapper[4771]: I1011 10:57:13.415610 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-combined-ca-bundle\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.432370 master-1 kubenswrapper[4771]: I1011 10:57:13.432295 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbq4h\" (UniqueName: \"kubernetes.io/projected/486db0a3-f081-43d5-b20d-d7386531632e-kube-api-access-lbq4h\") pod \"nova-metadata-1\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " pod="openstack/nova-metadata-1" Oct 11 10:57:13.436580 master-1 kubenswrapper[4771]: I1011 10:57:13.436504 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-scheduler-1" Oct 11 10:57:13.546284 master-1 kubenswrapper[4771]: I1011 10:57:13.543485 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:57:13.834472 master-1 kubenswrapper[4771]: I1011 10:57:13.833970 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"9a279d25-518e-4a12-8b75-e3781fb22f05","Type":"ContainerDied","Data":"26bb9eb9589e069594c3ca946945eba89d16e626b603cd344d3ddae2ac0f7ded"} Oct 11 10:57:13.834472 master-1 kubenswrapper[4771]: I1011 10:57:13.834060 4771 scope.go:117] "RemoveContainer" containerID="1856a76e1ae1c62aaea063758c5bd874b8a3b66130c9fe8eb095d5cef903001f" Oct 11 10:57:13.834472 master-1 kubenswrapper[4771]: I1011 10:57:13.834391 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:57:13.870817 master-1 kubenswrapper[4771]: I1011 10:57:13.870748 4771 scope.go:117] "RemoveContainer" containerID="06b1b45211e1a6439784f737849060952f6c89a6f37d2a3b15906929efc29bff" Oct 11 10:57:13.897525 master-1 kubenswrapper[4771]: I1011 10:57:13.897439 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:13.903733 master-1 kubenswrapper[4771]: I1011 10:57:13.903665 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:13.903856 master-1 kubenswrapper[4771]: I1011 10:57:13.903735 4771 scope.go:117] "RemoveContainer" containerID="dd5ef923fe19c8499d59b13d67599b3bf99730453e82ff9eb110343bd5333a66" Oct 11 10:57:13.932778 master-1 kubenswrapper[4771]: I1011 10:57:13.932714 4771 scope.go:117] "RemoveContainer" containerID="329f7fd8063c9aaba98ccaa627d41506880615f2c94e9a537c2fbe8e686bc8be" Oct 11 10:57:13.933840 master-1 kubenswrapper[4771]: I1011 10:57:13.933774 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:13.936549 master-1 kubenswrapper[4771]: I1011 10:57:13.936509 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:57:13.939457 master-1 kubenswrapper[4771]: I1011 10:57:13.939421 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 10:57:13.939773 master-1 kubenswrapper[4771]: I1011 10:57:13.939727 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ceilometer-internal-svc" Oct 11 10:57:13.939981 master-1 kubenswrapper[4771]: I1011 10:57:13.939936 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 10:57:13.955126 master-1 kubenswrapper[4771]: I1011 10:57:13.955038 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:14.028579 master-1 kubenswrapper[4771]: I1011 10:57:14.028506 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:14.033692 master-1 kubenswrapper[4771]: I1011 10:57:14.033641 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-config-data\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.033787 master-1 kubenswrapper[4771]: I1011 10:57:14.033759 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.033864 master-1 kubenswrapper[4771]: I1011 10:57:14.033802 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-run-httpd\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.033864 master-1 kubenswrapper[4771]: I1011 10:57:14.033822 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-scripts\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.033864 master-1 kubenswrapper[4771]: I1011 10:57:14.033848 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.034023 master-1 kubenswrapper[4771]: I1011 10:57:14.033872 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-log-httpd\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.034023 master-1 kubenswrapper[4771]: I1011 10:57:14.033908 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.034023 master-1 kubenswrapper[4771]: I1011 10:57:14.033928 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmlq6\" (UniqueName: \"kubernetes.io/projected/e85bacc4-2a43-4bc5-acc7-67f930ed6331-kube-api-access-tmlq6\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.135643 master-1 kubenswrapper[4771]: I1011 10:57:14.135580 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.135991 master-1 kubenswrapper[4771]: I1011 10:57:14.135663 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-run-httpd\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.135991 master-1 kubenswrapper[4771]: I1011 10:57:14.135685 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-scripts\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.135991 master-1 kubenswrapper[4771]: I1011 10:57:14.135735 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.135991 master-1 kubenswrapper[4771]: I1011 10:57:14.135771 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-log-httpd\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.135991 master-1 kubenswrapper[4771]: I1011 10:57:14.135830 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.135991 master-1 kubenswrapper[4771]: I1011 10:57:14.135853 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-tmlq6\" (UniqueName: \"kubernetes.io/projected/e85bacc4-2a43-4bc5-acc7-67f930ed6331-kube-api-access-tmlq6\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.137433 master-1 kubenswrapper[4771]: I1011 10:57:14.136575 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-config-data\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.141638 master-1 kubenswrapper[4771]: I1011 10:57:14.138258 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-log-httpd\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.141638 master-1 kubenswrapper[4771]: I1011 10:57:14.138280 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-run-httpd\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.144437 master-1 kubenswrapper[4771]: I1011 10:57:14.141882 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.144437 master-1 kubenswrapper[4771]: I1011 10:57:14.143478 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-config-data\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.144437 master-1 kubenswrapper[4771]: I1011 10:57:14.143759 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.144437 master-1 kubenswrapper[4771]: I1011 10:57:14.144128 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.144789 master-1 kubenswrapper[4771]: I1011 10:57:14.144435 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-scripts\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.165967 master-1 kubenswrapper[4771]: I1011 10:57:14.165897 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmlq6\" (UniqueName: \"kubernetes.io/projected/e85bacc4-2a43-4bc5-acc7-67f930ed6331-kube-api-access-tmlq6\") pod \"ceilometer-0\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " pod="openstack/ceilometer-0" Oct 11 10:57:14.260527 master-1 kubenswrapper[4771]: I1011 10:57:14.260409 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:57:14.468470 master-1 kubenswrapper[4771]: I1011 10:57:14.468427 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1596746b-25ca-487a-9e49-93e532f2838b" path="/var/lib/kubelet/pods/1596746b-25ca-487a-9e49-93e532f2838b/volumes" Oct 11 10:57:14.471597 master-1 kubenswrapper[4771]: I1011 10:57:14.471551 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a279d25-518e-4a12-8b75-e3781fb22f05" path="/var/lib/kubelet/pods/9a279d25-518e-4a12-8b75-e3781fb22f05/volumes" Oct 11 10:57:14.722263 master-1 kubenswrapper[4771]: I1011 10:57:14.721834 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:14.742061 master-1 kubenswrapper[4771]: W1011 10:57:14.741968 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode85bacc4_2a43_4bc5_acc7_67f930ed6331.slice/crio-be2095abce3f4ac809c0e84a1273a55d460312bdbba18373f246c0e8a9de1c8d WatchSource:0}: Error finding container be2095abce3f4ac809c0e84a1273a55d460312bdbba18373f246c0e8a9de1c8d: Status 404 returned error can't find the container with id be2095abce3f4ac809c0e84a1273a55d460312bdbba18373f246c0e8a9de1c8d Oct 11 10:57:14.858221 master-1 kubenswrapper[4771]: I1011 10:57:14.858139 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"486db0a3-f081-43d5-b20d-d7386531632e","Type":"ContainerStarted","Data":"80f0dd7f7bf3c1c4412d1e7a39722e2cd092b7c0f670f349af91c500d917aa10"} Oct 11 10:57:14.858221 master-1 kubenswrapper[4771]: I1011 10:57:14.858217 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"486db0a3-f081-43d5-b20d-d7386531632e","Type":"ContainerStarted","Data":"06a1bb6aa8f352b55c65006ce50a6d509e8dff8e484bf4f522e69ff0c42ae932"} Oct 11 10:57:14.858546 master-1 kubenswrapper[4771]: I1011 10:57:14.858239 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"486db0a3-f081-43d5-b20d-d7386531632e","Type":"ContainerStarted","Data":"5409d9254a9ac28cc3c943a6262472ba07dc81056b84eeb207f5cc4057ceaafd"} Oct 11 10:57:14.859739 master-1 kubenswrapper[4771]: I1011 10:57:14.859692 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerStarted","Data":"be2095abce3f4ac809c0e84a1273a55d460312bdbba18373f246c0e8a9de1c8d"} Oct 11 10:57:14.883845 master-1 kubenswrapper[4771]: I1011 10:57:14.883700 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-metadata-1" podStartSLOduration=1.8836824170000002 podStartE2EDuration="1.883682417s" podCreationTimestamp="2025-10-11 10:57:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:57:14.880832244 +0000 UTC m=+1866.855058685" watchObservedRunningTime="2025-10-11 10:57:14.883682417 +0000 UTC m=+1866.857908858" Oct 11 10:57:15.874075 master-1 kubenswrapper[4771]: I1011 10:57:15.873944 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerStarted","Data":"17e2191a3f20167201f2c27d474f69eb8338b55a6d88728e3892cdd495c418f4"} Oct 11 10:57:16.000877 master-1 kubenswrapper[4771]: I1011 10:57:16.000797 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:16.898421 master-1 kubenswrapper[4771]: I1011 10:57:16.897628 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerStarted","Data":"ebd23fc1f4d62b1e39557d15a1b710afe0ddaa7347500388c4a14b187788b9df"} Oct 11 10:57:16.898421 master-1 kubenswrapper[4771]: I1011 10:57:16.897736 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerStarted","Data":"7e99c868e28ca53790248959b5b2de659cb6170102635e6ace2cabc6e6703b85"} Oct 11 10:57:17.117599 master-1 kubenswrapper[4771]: I1011 10:57:17.117433 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:17.145509 master-1 kubenswrapper[4771]: I1011 10:57:17.145438 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:17.930414 master-1 kubenswrapper[4771]: I1011 10:57:17.930312 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-cell1-novncproxy-0" Oct 11 10:57:18.288342 master-1 kubenswrapper[4771]: I1011 10:57:18.288263 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell1-cell-mapping-bwgtz"] Oct 11 10:57:18.291318 master-1 kubenswrapper[4771]: I1011 10:57:18.291008 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.293974 master-1 kubenswrapper[4771]: I1011 10:57:18.293879 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell1-manage-config-data" Oct 11 10:57:18.299502 master-1 kubenswrapper[4771]: I1011 10:57:18.299432 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell1-manage-scripts" Oct 11 10:57:18.313266 master-1 kubenswrapper[4771]: I1011 10:57:18.313208 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-cell1-host-discover-5f556"] Oct 11 10:57:18.315826 master-1 kubenswrapper[4771]: I1011 10:57:18.315768 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.326266 master-1 kubenswrapper[4771]: I1011 10:57:18.326201 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-cell-mapping-bwgtz"] Oct 11 10:57:18.342203 master-1 kubenswrapper[4771]: I1011 10:57:18.342100 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-host-discover-5f556"] Oct 11 10:57:18.435658 master-1 kubenswrapper[4771]: I1011 10:57:18.435561 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-scheduler-1" Oct 11 10:57:18.454852 master-1 kubenswrapper[4771]: I1011 10:57:18.454782 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-combined-ca-bundle\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.454852 master-1 kubenswrapper[4771]: I1011 10:57:18.454839 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-config-data\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.455193 master-1 kubenswrapper[4771]: I1011 10:57:18.454907 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7dq\" (UniqueName: \"kubernetes.io/projected/ae49cc63-d351-440f-9334-4ef2550565a2-kube-api-access-6d7dq\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.455193 master-1 kubenswrapper[4771]: I1011 10:57:18.454943 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-combined-ca-bundle\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.455193 master-1 kubenswrapper[4771]: I1011 10:57:18.454969 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-scripts\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.455193 master-1 kubenswrapper[4771]: I1011 10:57:18.454990 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glspf\" (UniqueName: \"kubernetes.io/projected/709c362a-6ace-46bf-9f94-86852f78f6f2-kube-api-access-glspf\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.455193 master-1 kubenswrapper[4771]: I1011 10:57:18.455014 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-scripts\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.455193 master-1 kubenswrapper[4771]: I1011 10:57:18.455034 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-config-data\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.466902 master-1 kubenswrapper[4771]: I1011 10:57:18.466832 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-scheduler-1" Oct 11 10:57:18.545195 master-1 kubenswrapper[4771]: I1011 10:57:18.545126 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-metadata-1" Oct 11 10:57:18.545195 master-1 kubenswrapper[4771]: I1011 10:57:18.545194 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-metadata-1" Oct 11 10:57:18.556821 master-1 kubenswrapper[4771]: I1011 10:57:18.556440 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7dq\" (UniqueName: \"kubernetes.io/projected/ae49cc63-d351-440f-9334-4ef2550565a2-kube-api-access-6d7dq\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.557459 master-1 kubenswrapper[4771]: I1011 10:57:18.557408 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-combined-ca-bundle\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.557567 master-1 kubenswrapper[4771]: I1011 10:57:18.557475 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-scripts\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.557567 master-1 kubenswrapper[4771]: I1011 10:57:18.557513 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-glspf\" (UniqueName: \"kubernetes.io/projected/709c362a-6ace-46bf-9f94-86852f78f6f2-kube-api-access-glspf\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.557567 master-1 kubenswrapper[4771]: I1011 10:57:18.557546 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-scripts\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.557770 master-1 kubenswrapper[4771]: I1011 10:57:18.557574 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-config-data\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.557770 master-1 kubenswrapper[4771]: I1011 10:57:18.557676 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-combined-ca-bundle\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.557770 master-1 kubenswrapper[4771]: I1011 10:57:18.557717 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-config-data\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.562876 master-1 kubenswrapper[4771]: I1011 10:57:18.562670 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-config-data\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.564809 master-1 kubenswrapper[4771]: I1011 10:57:18.564560 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-config-data\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.566938 master-1 kubenswrapper[4771]: I1011 10:57:18.566820 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-combined-ca-bundle\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.567098 master-1 kubenswrapper[4771]: I1011 10:57:18.567059 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-combined-ca-bundle\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.567980 master-1 kubenswrapper[4771]: I1011 10:57:18.567805 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-scripts\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.569230 master-1 kubenswrapper[4771]: I1011 10:57:18.569040 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-scripts\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.590401 master-1 kubenswrapper[4771]: I1011 10:57:18.588561 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7dq\" (UniqueName: \"kubernetes.io/projected/ae49cc63-d351-440f-9334-4ef2550565a2-kube-api-access-6d7dq\") pod \"nova-cell1-host-discover-5f556\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.598159 master-1 kubenswrapper[4771]: I1011 10:57:18.598115 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-glspf\" (UniqueName: \"kubernetes.io/projected/709c362a-6ace-46bf-9f94-86852f78f6f2-kube-api-access-glspf\") pod \"nova-cell1-cell-mapping-bwgtz\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.617482 master-1 kubenswrapper[4771]: I1011 10:57:18.617405 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:18.644727 master-1 kubenswrapper[4771]: I1011 10:57:18.644643 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:18.920485 master-1 kubenswrapper[4771]: I1011 10:57:18.920413 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerStarted","Data":"c214200e01db88cd82a2be7612b0009a76284d886cfd5582dbd4340ef9a3bf14"} Oct 11 10:57:18.921370 master-1 kubenswrapper[4771]: I1011 10:57:18.921321 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="ceilometer-central-agent" containerID="cri-o://17e2191a3f20167201f2c27d474f69eb8338b55a6d88728e3892cdd495c418f4" gracePeriod=30 Oct 11 10:57:18.921471 master-1 kubenswrapper[4771]: I1011 10:57:18.921431 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="proxy-httpd" containerID="cri-o://c214200e01db88cd82a2be7612b0009a76284d886cfd5582dbd4340ef9a3bf14" gracePeriod=30 Oct 11 10:57:18.921635 master-1 kubenswrapper[4771]: I1011 10:57:18.921601 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="ceilometer-notification-agent" containerID="cri-o://7e99c868e28ca53790248959b5b2de659cb6170102635e6ace2cabc6e6703b85" gracePeriod=30 Oct 11 10:57:18.921701 master-1 kubenswrapper[4771]: I1011 10:57:18.921681 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="sg-core" containerID="cri-o://ebd23fc1f4d62b1e39557d15a1b710afe0ddaa7347500388c4a14b187788b9df" gracePeriod=30 Oct 11 10:57:18.953144 master-1 kubenswrapper[4771]: I1011 10:57:18.953102 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-scheduler-1" Oct 11 10:57:18.954492 master-1 kubenswrapper[4771]: I1011 10:57:18.954438 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=2.899843392 podStartE2EDuration="5.954421811s" podCreationTimestamp="2025-10-11 10:57:13 +0000 UTC" firstStartedPulling="2025-10-11 10:57:14.746498557 +0000 UTC m=+1866.720725008" lastFinishedPulling="2025-10-11 10:57:17.801076956 +0000 UTC m=+1869.775303427" observedRunningTime="2025-10-11 10:57:18.953456083 +0000 UTC m=+1870.927682564" watchObservedRunningTime="2025-10-11 10:57:18.954421811 +0000 UTC m=+1870.928648252" Oct 11 10:57:19.066672 master-1 kubenswrapper[4771]: I1011 10:57:19.066618 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-cell-mapping-bwgtz"] Oct 11 10:57:19.201255 master-1 kubenswrapper[4771]: I1011 10:57:19.201211 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-cell1-host-discover-5f556"] Oct 11 10:57:19.223926 master-1 kubenswrapper[4771]: W1011 10:57:19.223884 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae49cc63_d351_440f_9334_4ef2550565a2.slice/crio-bf7cd872d077a706534449020eb602438e4892d62ec0c9d5010cb1330868aeea WatchSource:0}: Error finding container bf7cd872d077a706534449020eb602438e4892d62ec0c9d5010cb1330868aeea: Status 404 returned error can't find the container with id bf7cd872d077a706534449020eb602438e4892d62ec0c9d5010cb1330868aeea Oct 11 10:57:19.936390 master-1 kubenswrapper[4771]: I1011 10:57:19.936163 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-host-discover-5f556" event={"ID":"ae49cc63-d351-440f-9334-4ef2550565a2","Type":"ContainerStarted","Data":"48e35ef26a01bac7444e96fa2a9fa3fe07bd9eb6b20913ec8c1c945288cc11bc"} Oct 11 10:57:19.936390 master-1 kubenswrapper[4771]: I1011 10:57:19.936253 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-host-discover-5f556" event={"ID":"ae49cc63-d351-440f-9334-4ef2550565a2","Type":"ContainerStarted","Data":"bf7cd872d077a706534449020eb602438e4892d62ec0c9d5010cb1330868aeea"} Oct 11 10:57:19.939417 master-1 kubenswrapper[4771]: I1011 10:57:19.939318 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-cell-mapping-bwgtz" event={"ID":"709c362a-6ace-46bf-9f94-86852f78f6f2","Type":"ContainerStarted","Data":"99d58d9d6b8b62fa18ae8ba7508466dad2a9761e505b9274423ecba095a9de64"} Oct 11 10:57:19.939498 master-1 kubenswrapper[4771]: I1011 10:57:19.939425 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-cell-mapping-bwgtz" event={"ID":"709c362a-6ace-46bf-9f94-86852f78f6f2","Type":"ContainerStarted","Data":"4cefc06e3826c53b2bddfc65675ffb15401ad9ff18e58b5e2736c262f90fe5e5"} Oct 11 10:57:19.947373 master-1 kubenswrapper[4771]: I1011 10:57:19.947170 4771 generic.go:334] "Generic (PLEG): container finished" podID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerID="c214200e01db88cd82a2be7612b0009a76284d886cfd5582dbd4340ef9a3bf14" exitCode=0 Oct 11 10:57:19.947373 master-1 kubenswrapper[4771]: I1011 10:57:19.947220 4771 generic.go:334] "Generic (PLEG): container finished" podID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerID="ebd23fc1f4d62b1e39557d15a1b710afe0ddaa7347500388c4a14b187788b9df" exitCode=2 Oct 11 10:57:19.947373 master-1 kubenswrapper[4771]: I1011 10:57:19.947238 4771 generic.go:334] "Generic (PLEG): container finished" podID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerID="7e99c868e28ca53790248959b5b2de659cb6170102635e6ace2cabc6e6703b85" exitCode=0 Oct 11 10:57:19.947373 master-1 kubenswrapper[4771]: I1011 10:57:19.947271 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerDied","Data":"c214200e01db88cd82a2be7612b0009a76284d886cfd5582dbd4340ef9a3bf14"} Oct 11 10:57:19.947373 master-1 kubenswrapper[4771]: I1011 10:57:19.947336 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerDied","Data":"ebd23fc1f4d62b1e39557d15a1b710afe0ddaa7347500388c4a14b187788b9df"} Oct 11 10:57:19.947585 master-1 kubenswrapper[4771]: I1011 10:57:19.947397 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerDied","Data":"7e99c868e28ca53790248959b5b2de659cb6170102635e6ace2cabc6e6703b85"} Oct 11 10:57:19.975456 master-1 kubenswrapper[4771]: I1011 10:57:19.975284 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell1-host-discover-5f556" podStartSLOduration=1.975250991 podStartE2EDuration="1.975250991s" podCreationTimestamp="2025-10-11 10:57:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:57:19.970171663 +0000 UTC m=+1871.944398154" watchObservedRunningTime="2025-10-11 10:57:19.975250991 +0000 UTC m=+1871.949477502" Oct 11 10:57:20.000678 master-1 kubenswrapper[4771]: I1011 10:57:20.000568 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-cell1-cell-mapping-bwgtz" podStartSLOduration=2.000542879 podStartE2EDuration="2.000542879s" podCreationTimestamp="2025-10-11 10:57:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:57:19.995153942 +0000 UTC m=+1871.969380433" watchObservedRunningTime="2025-10-11 10:57:20.000542879 +0000 UTC m=+1871.974769360" Oct 11 10:57:20.822661 master-1 kubenswrapper[4771]: I1011 10:57:20.822603 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-79cbf74f6f-j7kt4"] Oct 11 10:57:20.822901 master-1 kubenswrapper[4771]: I1011 10:57:20.822868 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" podUID="a23d84be-f5ab-4261-9ba2-d94aaf104a59" containerName="dnsmasq-dns" containerID="cri-o://12c6ff03be76828491f921afc8c9ec6e58880687794d58647b68e34022915241" gracePeriod=10 Oct 11 10:57:20.861764 master-1 kubenswrapper[4771]: I1011 10:57:20.861294 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-6cb9b8c955-b5qwc"] Oct 11 10:57:20.864023 master-1 kubenswrapper[4771]: I1011 10:57:20.863974 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:20.881943 master-1 kubenswrapper[4771]: I1011 10:57:20.881854 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-6cb9b8c955-b5qwc"] Oct 11 10:57:20.921978 master-1 kubenswrapper[4771]: I1011 10:57:20.920700 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/396249ad-10d3-48d9-ba43-46df789198c9-kube-api-access-897k7\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:20.921978 master-1 kubenswrapper[4771]: I1011 10:57:20.920791 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-config\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:20.921978 master-1 kubenswrapper[4771]: I1011 10:57:20.920857 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-nb\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:20.921978 master-1 kubenswrapper[4771]: I1011 10:57:20.920907 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-svc\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:20.921978 master-1 kubenswrapper[4771]: I1011 10:57:20.920924 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-sb\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:20.921978 master-1 kubenswrapper[4771]: I1011 10:57:20.920983 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-swift-storage-0\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:20.958133 master-1 kubenswrapper[4771]: I1011 10:57:20.958083 4771 generic.go:334] "Generic (PLEG): container finished" podID="a23d84be-f5ab-4261-9ba2-d94aaf104a59" containerID="12c6ff03be76828491f921afc8c9ec6e58880687794d58647b68e34022915241" exitCode=0 Oct 11 10:57:20.958250 master-1 kubenswrapper[4771]: I1011 10:57:20.958197 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" event={"ID":"a23d84be-f5ab-4261-9ba2-d94aaf104a59","Type":"ContainerDied","Data":"12c6ff03be76828491f921afc8c9ec6e58880687794d58647b68e34022915241"} Oct 11 10:57:20.961726 master-1 kubenswrapper[4771]: I1011 10:57:20.961690 4771 generic.go:334] "Generic (PLEG): container finished" podID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerID="17e2191a3f20167201f2c27d474f69eb8338b55a6d88728e3892cdd495c418f4" exitCode=0 Oct 11 10:57:20.961787 master-1 kubenswrapper[4771]: I1011 10:57:20.961758 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerDied","Data":"17e2191a3f20167201f2c27d474f69eb8338b55a6d88728e3892cdd495c418f4"} Oct 11 10:57:21.021984 master-1 kubenswrapper[4771]: I1011 10:57:21.021937 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/396249ad-10d3-48d9-ba43-46df789198c9-kube-api-access-897k7\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.022515 master-1 kubenswrapper[4771]: I1011 10:57:21.021998 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-config\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.022738 master-1 kubenswrapper[4771]: I1011 10:57:21.022625 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-nb\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.024183 master-1 kubenswrapper[4771]: I1011 10:57:21.022834 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-svc\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.024183 master-1 kubenswrapper[4771]: I1011 10:57:21.022885 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-sb\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.024183 master-1 kubenswrapper[4771]: I1011 10:57:21.023064 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-config\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.024183 master-1 kubenswrapper[4771]: I1011 10:57:21.023112 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-swift-storage-0\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.024183 master-1 kubenswrapper[4771]: I1011 10:57:21.023747 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-nb\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.024472 master-1 kubenswrapper[4771]: I1011 10:57:21.024182 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-svc\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.024472 master-1 kubenswrapper[4771]: I1011 10:57:21.024301 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-swift-storage-0\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.024629 master-1 kubenswrapper[4771]: I1011 10:57:21.024589 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-sb\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.063185 master-1 kubenswrapper[4771]: I1011 10:57:21.063127 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/396249ad-10d3-48d9-ba43-46df789198c9-kube-api-access-897k7\") pod \"dnsmasq-dns-6cb9b8c955-b5qwc\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.321469 master-1 kubenswrapper[4771]: I1011 10:57:21.321410 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:21.529427 master-1 kubenswrapper[4771]: I1011 10:57:21.528900 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:57:21.538170 master-1 kubenswrapper[4771]: I1011 10:57:21.537724 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:57:21.640123 master-1 kubenswrapper[4771]: I1011 10:57:21.640063 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-combined-ca-bundle\") pod \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " Oct 11 10:57:21.640330 master-1 kubenswrapper[4771]: I1011 10:57:21.640140 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-scripts\") pod \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " Oct 11 10:57:21.640330 master-1 kubenswrapper[4771]: I1011 10:57:21.640188 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7n6h\" (UniqueName: \"kubernetes.io/projected/a23d84be-f5ab-4261-9ba2-d94aaf104a59-kube-api-access-k7n6h\") pod \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " Oct 11 10:57:21.640330 master-1 kubenswrapper[4771]: I1011 10:57:21.640235 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-nb\") pod \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " Oct 11 10:57:21.640330 master-1 kubenswrapper[4771]: I1011 10:57:21.640263 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-svc\") pod \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " Oct 11 10:57:21.640494 master-1 kubenswrapper[4771]: I1011 10:57:21.640421 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-log-httpd\") pod \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " Oct 11 10:57:21.640494 master-1 kubenswrapper[4771]: I1011 10:57:21.640449 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmlq6\" (UniqueName: \"kubernetes.io/projected/e85bacc4-2a43-4bc5-acc7-67f930ed6331-kube-api-access-tmlq6\") pod \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " Oct 11 10:57:21.640494 master-1 kubenswrapper[4771]: I1011 10:57:21.640475 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-config-data\") pod \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " Oct 11 10:57:21.640591 master-1 kubenswrapper[4771]: I1011 10:57:21.640509 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-swift-storage-0\") pod \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " Oct 11 10:57:21.640591 master-1 kubenswrapper[4771]: I1011 10:57:21.640537 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-ceilometer-tls-certs\") pod \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " Oct 11 10:57:21.640735 master-1 kubenswrapper[4771]: I1011 10:57:21.640706 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-sb\") pod \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " Oct 11 10:57:21.640794 master-1 kubenswrapper[4771]: I1011 10:57:21.640774 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-run-httpd\") pod \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " Oct 11 10:57:21.640831 master-1 kubenswrapper[4771]: I1011 10:57:21.640802 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-sg-core-conf-yaml\") pod \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\" (UID: \"e85bacc4-2a43-4bc5-acc7-67f930ed6331\") " Oct 11 10:57:21.640867 master-1 kubenswrapper[4771]: I1011 10:57:21.640842 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-config\") pod \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\" (UID: \"a23d84be-f5ab-4261-9ba2-d94aaf104a59\") " Oct 11 10:57:21.643117 master-1 kubenswrapper[4771]: I1011 10:57:21.642822 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "e85bacc4-2a43-4bc5-acc7-67f930ed6331" (UID: "e85bacc4-2a43-4bc5-acc7-67f930ed6331"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:57:21.643117 master-1 kubenswrapper[4771]: I1011 10:57:21.642991 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "e85bacc4-2a43-4bc5-acc7-67f930ed6331" (UID: "e85bacc4-2a43-4bc5-acc7-67f930ed6331"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:57:21.646146 master-1 kubenswrapper[4771]: I1011 10:57:21.646099 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23d84be-f5ab-4261-9ba2-d94aaf104a59-kube-api-access-k7n6h" (OuterVolumeSpecName: "kube-api-access-k7n6h") pod "a23d84be-f5ab-4261-9ba2-d94aaf104a59" (UID: "a23d84be-f5ab-4261-9ba2-d94aaf104a59"). InnerVolumeSpecName "kube-api-access-k7n6h". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:21.646799 master-1 kubenswrapper[4771]: I1011 10:57:21.646762 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85bacc4-2a43-4bc5-acc7-67f930ed6331-kube-api-access-tmlq6" (OuterVolumeSpecName: "kube-api-access-tmlq6") pod "e85bacc4-2a43-4bc5-acc7-67f930ed6331" (UID: "e85bacc4-2a43-4bc5-acc7-67f930ed6331"). InnerVolumeSpecName "kube-api-access-tmlq6". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:21.647064 master-1 kubenswrapper[4771]: I1011 10:57:21.647029 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-scripts" (OuterVolumeSpecName: "scripts") pod "e85bacc4-2a43-4bc5-acc7-67f930ed6331" (UID: "e85bacc4-2a43-4bc5-acc7-67f930ed6331"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:21.684852 master-1 kubenswrapper[4771]: I1011 10:57:21.684786 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-nb" (OuterVolumeSpecName: "ovsdbserver-nb") pod "a23d84be-f5ab-4261-9ba2-d94aaf104a59" (UID: "a23d84be-f5ab-4261-9ba2-d94aaf104a59"). InnerVolumeSpecName "ovsdbserver-nb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:57:21.685149 master-1 kubenswrapper[4771]: I1011 10:57:21.685118 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "a23d84be-f5ab-4261-9ba2-d94aaf104a59" (UID: "a23d84be-f5ab-4261-9ba2-d94aaf104a59"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:57:21.686515 master-1 kubenswrapper[4771]: I1011 10:57:21.686468 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-ceilometer-tls-certs" (OuterVolumeSpecName: "ceilometer-tls-certs") pod "e85bacc4-2a43-4bc5-acc7-67f930ed6331" (UID: "e85bacc4-2a43-4bc5-acc7-67f930ed6331"). InnerVolumeSpecName "ceilometer-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:21.686957 master-1 kubenswrapper[4771]: I1011 10:57:21.686912 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-config" (OuterVolumeSpecName: "config") pod "a23d84be-f5ab-4261-9ba2-d94aaf104a59" (UID: "a23d84be-f5ab-4261-9ba2-d94aaf104a59"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:57:21.687751 master-1 kubenswrapper[4771]: I1011 10:57:21.687702 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "e85bacc4-2a43-4bc5-acc7-67f930ed6331" (UID: "e85bacc4-2a43-4bc5-acc7-67f930ed6331"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:21.697748 master-1 kubenswrapper[4771]: I1011 10:57:21.697716 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-swift-storage-0" (OuterVolumeSpecName: "dns-swift-storage-0") pod "a23d84be-f5ab-4261-9ba2-d94aaf104a59" (UID: "a23d84be-f5ab-4261-9ba2-d94aaf104a59"). InnerVolumeSpecName "dns-swift-storage-0". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:57:21.702223 master-1 kubenswrapper[4771]: I1011 10:57:21.702189 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-sb" (OuterVolumeSpecName: "ovsdbserver-sb") pod "a23d84be-f5ab-4261-9ba2-d94aaf104a59" (UID: "a23d84be-f5ab-4261-9ba2-d94aaf104a59"). InnerVolumeSpecName "ovsdbserver-sb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 10:57:21.727130 master-1 kubenswrapper[4771]: I1011 10:57:21.727069 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "e85bacc4-2a43-4bc5-acc7-67f930ed6331" (UID: "e85bacc4-2a43-4bc5-acc7-67f930ed6331"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:21.743278 master-1 kubenswrapper[4771]: I1011 10:57:21.743220 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743387 master-1 kubenswrapper[4771]: I1011 10:57:21.743300 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743387 master-1 kubenswrapper[4771]: I1011 10:57:21.743320 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-config\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743387 master-1 kubenswrapper[4771]: I1011 10:57:21.743336 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743387 master-1 kubenswrapper[4771]: I1011 10:57:21.743375 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743505 master-1 kubenswrapper[4771]: I1011 10:57:21.743393 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-k7n6h\" (UniqueName: \"kubernetes.io/projected/a23d84be-f5ab-4261-9ba2-d94aaf104a59-kube-api-access-k7n6h\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743505 master-1 kubenswrapper[4771]: I1011 10:57:21.743411 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-nb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743505 master-1 kubenswrapper[4771]: I1011 10:57:21.743426 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743505 master-1 kubenswrapper[4771]: I1011 10:57:21.743442 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/e85bacc4-2a43-4bc5-acc7-67f930ed6331-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743505 master-1 kubenswrapper[4771]: I1011 10:57:21.743460 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-tmlq6\" (UniqueName: \"kubernetes.io/projected/e85bacc4-2a43-4bc5-acc7-67f930ed6331-kube-api-access-tmlq6\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743505 master-1 kubenswrapper[4771]: I1011 10:57:21.743477 4771 reconciler_common.go:293] "Volume detached for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-dns-swift-storage-0\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743505 master-1 kubenswrapper[4771]: I1011 10:57:21.743492 4771 reconciler_common.go:293] "Volume detached for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-ceilometer-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.743505 master-1 kubenswrapper[4771]: I1011 10:57:21.743507 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/a23d84be-f5ab-4261-9ba2-d94aaf104a59-ovsdbserver-sb\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.759802 master-1 kubenswrapper[4771]: I1011 10:57:21.759728 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-config-data" (OuterVolumeSpecName: "config-data") pod "e85bacc4-2a43-4bc5-acc7-67f930ed6331" (UID: "e85bacc4-2a43-4bc5-acc7-67f930ed6331"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:21.798539 master-1 kubenswrapper[4771]: I1011 10:57:21.798503 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-6cb9b8c955-b5qwc"] Oct 11 10:57:21.845081 master-1 kubenswrapper[4771]: I1011 10:57:21.845056 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e85bacc4-2a43-4bc5-acc7-67f930ed6331-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:21.971220 master-1 kubenswrapper[4771]: I1011 10:57:21.971182 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" event={"ID":"396249ad-10d3-48d9-ba43-46df789198c9","Type":"ContainerStarted","Data":"a14cd6677525c65737d0849bb25554909c4ebb8c2b5761120df0ab99b361a3df"} Oct 11 10:57:21.973968 master-1 kubenswrapper[4771]: I1011 10:57:21.973942 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" event={"ID":"a23d84be-f5ab-4261-9ba2-d94aaf104a59","Type":"ContainerDied","Data":"b0d191f73463f5a71aeb190809caf8100724d2aeec1100c76a864a58130b5a3d"} Oct 11 10:57:21.974043 master-1 kubenswrapper[4771]: I1011 10:57:21.973954 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-79cbf74f6f-j7kt4" Oct 11 10:57:21.974043 master-1 kubenswrapper[4771]: I1011 10:57:21.973997 4771 scope.go:117] "RemoveContainer" containerID="12c6ff03be76828491f921afc8c9ec6e58880687794d58647b68e34022915241" Oct 11 10:57:21.978113 master-1 kubenswrapper[4771]: I1011 10:57:21.978047 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"e85bacc4-2a43-4bc5-acc7-67f930ed6331","Type":"ContainerDied","Data":"be2095abce3f4ac809c0e84a1273a55d460312bdbba18373f246c0e8a9de1c8d"} Oct 11 10:57:21.978160 master-1 kubenswrapper[4771]: I1011 10:57:21.978128 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:57:21.994926 master-1 kubenswrapper[4771]: I1011 10:57:21.994845 4771 scope.go:117] "RemoveContainer" containerID="14cbbf6abeb88d28f08a7099ac711df9a488bc85a2f7bd445bc229705a05a25b" Oct 11 10:57:22.035128 master-1 kubenswrapper[4771]: I1011 10:57:22.035082 4771 scope.go:117] "RemoveContainer" containerID="c214200e01db88cd82a2be7612b0009a76284d886cfd5582dbd4340ef9a3bf14" Oct 11 10:57:22.043561 master-1 kubenswrapper[4771]: I1011 10:57:22.043524 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-79cbf74f6f-j7kt4"] Oct 11 10:57:22.051896 master-1 kubenswrapper[4771]: I1011 10:57:22.051859 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-79cbf74f6f-j7kt4"] Oct 11 10:57:22.076703 master-1 kubenswrapper[4771]: I1011 10:57:22.076658 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:22.103789 master-1 kubenswrapper[4771]: I1011 10:57:22.103749 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:22.127659 master-1 kubenswrapper[4771]: I1011 10:57:22.127615 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:22.128196 master-1 kubenswrapper[4771]: E1011 10:57:22.128181 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a23d84be-f5ab-4261-9ba2-d94aaf104a59" containerName="init" Oct 11 10:57:22.128265 master-1 kubenswrapper[4771]: I1011 10:57:22.128255 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23d84be-f5ab-4261-9ba2-d94aaf104a59" containerName="init" Oct 11 10:57:22.128339 master-1 kubenswrapper[4771]: E1011 10:57:22.128327 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="ceilometer-notification-agent" Oct 11 10:57:22.128406 master-1 kubenswrapper[4771]: I1011 10:57:22.128396 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="ceilometer-notification-agent" Oct 11 10:57:22.128479 master-1 kubenswrapper[4771]: E1011 10:57:22.128470 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a23d84be-f5ab-4261-9ba2-d94aaf104a59" containerName="dnsmasq-dns" Oct 11 10:57:22.128530 master-1 kubenswrapper[4771]: I1011 10:57:22.128521 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23d84be-f5ab-4261-9ba2-d94aaf104a59" containerName="dnsmasq-dns" Oct 11 10:57:22.128584 master-1 kubenswrapper[4771]: E1011 10:57:22.128575 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="ceilometer-central-agent" Oct 11 10:57:22.128639 master-1 kubenswrapper[4771]: I1011 10:57:22.128630 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="ceilometer-central-agent" Oct 11 10:57:22.128694 master-1 kubenswrapper[4771]: E1011 10:57:22.128685 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="proxy-httpd" Oct 11 10:57:22.128743 master-1 kubenswrapper[4771]: I1011 10:57:22.128734 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="proxy-httpd" Oct 11 10:57:22.128796 master-1 kubenswrapper[4771]: E1011 10:57:22.128788 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="sg-core" Oct 11 10:57:22.128854 master-1 kubenswrapper[4771]: I1011 10:57:22.128845 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="sg-core" Oct 11 10:57:22.129032 master-1 kubenswrapper[4771]: I1011 10:57:22.129021 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="sg-core" Oct 11 10:57:22.129103 master-1 kubenswrapper[4771]: I1011 10:57:22.129094 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="ceilometer-central-agent" Oct 11 10:57:22.129166 master-1 kubenswrapper[4771]: I1011 10:57:22.129158 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a23d84be-f5ab-4261-9ba2-d94aaf104a59" containerName="dnsmasq-dns" Oct 11 10:57:22.129226 master-1 kubenswrapper[4771]: I1011 10:57:22.129217 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="ceilometer-notification-agent" Oct 11 10:57:22.129295 master-1 kubenswrapper[4771]: I1011 10:57:22.129285 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" containerName="proxy-httpd" Oct 11 10:57:22.131625 master-1 kubenswrapper[4771]: I1011 10:57:22.131612 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:57:22.141530 master-1 kubenswrapper[4771]: I1011 10:57:22.136616 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ceilometer-internal-svc" Oct 11 10:57:22.141530 master-1 kubenswrapper[4771]: I1011 10:57:22.136832 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 10:57:22.141530 master-1 kubenswrapper[4771]: I1011 10:57:22.136836 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 10:57:22.143496 master-1 kubenswrapper[4771]: I1011 10:57:22.143469 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:22.269189 master-1 kubenswrapper[4771]: I1011 10:57:22.269129 4771 scope.go:117] "RemoveContainer" containerID="ebd23fc1f4d62b1e39557d15a1b710afe0ddaa7347500388c4a14b187788b9df" Oct 11 10:57:22.275858 master-1 kubenswrapper[4771]: I1011 10:57:22.275783 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-scripts\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.276005 master-1 kubenswrapper[4771]: I1011 10:57:22.275907 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-run-httpd\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.276005 master-1 kubenswrapper[4771]: I1011 10:57:22.275984 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.276197 master-1 kubenswrapper[4771]: I1011 10:57:22.276173 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.276340 master-1 kubenswrapper[4771]: I1011 10:57:22.276212 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-log-httpd\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.276340 master-1 kubenswrapper[4771]: I1011 10:57:22.276249 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-config-data\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.276340 master-1 kubenswrapper[4771]: I1011 10:57:22.276315 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.276653 master-1 kubenswrapper[4771]: I1011 10:57:22.276351 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwhv2\" (UniqueName: \"kubernetes.io/projected/926f8cdc-bbf6-4328-8436-8428df0a679b-kube-api-access-xwhv2\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.295837 master-1 kubenswrapper[4771]: I1011 10:57:22.295713 4771 scope.go:117] "RemoveContainer" containerID="7e99c868e28ca53790248959b5b2de659cb6170102635e6ace2cabc6e6703b85" Oct 11 10:57:22.321615 master-1 kubenswrapper[4771]: I1011 10:57:22.321566 4771 scope.go:117] "RemoveContainer" containerID="17e2191a3f20167201f2c27d474f69eb8338b55a6d88728e3892cdd495c418f4" Oct 11 10:57:22.380191 master-1 kubenswrapper[4771]: I1011 10:57:22.379849 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-scripts\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.380191 master-1 kubenswrapper[4771]: I1011 10:57:22.379964 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-run-httpd\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.380191 master-1 kubenswrapper[4771]: I1011 10:57:22.380009 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.380191 master-1 kubenswrapper[4771]: I1011 10:57:22.380095 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.380191 master-1 kubenswrapper[4771]: I1011 10:57:22.380119 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-log-httpd\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.380191 master-1 kubenswrapper[4771]: I1011 10:57:22.380140 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-config-data\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.380191 master-1 kubenswrapper[4771]: I1011 10:57:22.380161 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.380191 master-1 kubenswrapper[4771]: I1011 10:57:22.380181 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-xwhv2\" (UniqueName: \"kubernetes.io/projected/926f8cdc-bbf6-4328-8436-8428df0a679b-kube-api-access-xwhv2\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.385484 master-1 kubenswrapper[4771]: I1011 10:57:22.380788 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-run-httpd\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.385484 master-1 kubenswrapper[4771]: I1011 10:57:22.381165 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-log-httpd\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.385484 master-1 kubenswrapper[4771]: I1011 10:57:22.385135 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-config-data\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.385779 master-1 kubenswrapper[4771]: I1011 10:57:22.385599 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.385779 master-1 kubenswrapper[4771]: I1011 10:57:22.385639 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.395444 master-1 kubenswrapper[4771]: I1011 10:57:22.389170 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-scripts\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.395444 master-1 kubenswrapper[4771]: I1011 10:57:22.391506 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.409726 master-1 kubenswrapper[4771]: I1011 10:57:22.409681 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwhv2\" (UniqueName: \"kubernetes.io/projected/926f8cdc-bbf6-4328-8436-8428df0a679b-kube-api-access-xwhv2\") pod \"ceilometer-0\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " pod="openstack/ceilometer-0" Oct 11 10:57:22.446306 master-1 kubenswrapper[4771]: I1011 10:57:22.446242 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23d84be-f5ab-4261-9ba2-d94aaf104a59" path="/var/lib/kubelet/pods/a23d84be-f5ab-4261-9ba2-d94aaf104a59/volumes" Oct 11 10:57:22.447163 master-1 kubenswrapper[4771]: I1011 10:57:22.447114 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e85bacc4-2a43-4bc5-acc7-67f930ed6331" path="/var/lib/kubelet/pods/e85bacc4-2a43-4bc5-acc7-67f930ed6331/volumes" Oct 11 10:57:22.582316 master-1 kubenswrapper[4771]: I1011 10:57:22.582262 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 10:57:22.992475 master-1 kubenswrapper[4771]: I1011 10:57:22.992288 4771 generic.go:334] "Generic (PLEG): container finished" podID="ae49cc63-d351-440f-9334-4ef2550565a2" containerID="48e35ef26a01bac7444e96fa2a9fa3fe07bd9eb6b20913ec8c1c945288cc11bc" exitCode=0 Oct 11 10:57:22.992727 master-1 kubenswrapper[4771]: I1011 10:57:22.992498 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-host-discover-5f556" event={"ID":"ae49cc63-d351-440f-9334-4ef2550565a2","Type":"ContainerDied","Data":"48e35ef26a01bac7444e96fa2a9fa3fe07bd9eb6b20913ec8c1c945288cc11bc"} Oct 11 10:57:22.995578 master-1 kubenswrapper[4771]: I1011 10:57:22.995549 4771 generic.go:334] "Generic (PLEG): container finished" podID="396249ad-10d3-48d9-ba43-46df789198c9" containerID="56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2" exitCode=0 Oct 11 10:57:22.995696 master-1 kubenswrapper[4771]: I1011 10:57:22.995591 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" event={"ID":"396249ad-10d3-48d9-ba43-46df789198c9","Type":"ContainerDied","Data":"56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2"} Oct 11 10:57:23.084455 master-1 kubenswrapper[4771]: I1011 10:57:23.084392 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 10:57:23.122371 master-1 kubenswrapper[4771]: W1011 10:57:23.122313 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod926f8cdc_bbf6_4328_8436_8428df0a679b.slice/crio-2a75cd52ad1f72be5ccb56e9952a02a3c2bfd8c3f845acfbe551f4d25daeffc2 WatchSource:0}: Error finding container 2a75cd52ad1f72be5ccb56e9952a02a3c2bfd8c3f845acfbe551f4d25daeffc2: Status 404 returned error can't find the container with id 2a75cd52ad1f72be5ccb56e9952a02a3c2bfd8c3f845acfbe551f4d25daeffc2 Oct 11 10:57:23.545369 master-1 kubenswrapper[4771]: I1011 10:57:23.545199 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-metadata-1" Oct 11 10:57:23.545875 master-1 kubenswrapper[4771]: I1011 10:57:23.545855 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-metadata-1" Oct 11 10:57:24.016517 master-1 kubenswrapper[4771]: I1011 10:57:24.016420 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" event={"ID":"396249ad-10d3-48d9-ba43-46df789198c9","Type":"ContainerStarted","Data":"6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760"} Oct 11 10:57:24.016850 master-1 kubenswrapper[4771]: I1011 10:57:24.016534 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:24.018814 master-1 kubenswrapper[4771]: I1011 10:57:24.018498 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerStarted","Data":"2a75cd52ad1f72be5ccb56e9952a02a3c2bfd8c3f845acfbe551f4d25daeffc2"} Oct 11 10:57:24.055669 master-1 kubenswrapper[4771]: I1011 10:57:24.055509 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" podStartSLOduration=4.055446081 podStartE2EDuration="4.055446081s" podCreationTimestamp="2025-10-11 10:57:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:57:24.049714924 +0000 UTC m=+1876.023941405" watchObservedRunningTime="2025-10-11 10:57:24.055446081 +0000 UTC m=+1876.029672562" Oct 11 10:57:24.513726 master-1 kubenswrapper[4771]: I1011 10:57:24.513639 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:24.565677 master-1 kubenswrapper[4771]: I1011 10:57:24.565565 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-metadata-1" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-metadata" probeResult="failure" output="Get \"https://10.129.0.168:8775/\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:57:24.566015 master-1 kubenswrapper[4771]: I1011 10:57:24.565927 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-metadata-1" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-log" probeResult="failure" output="Get \"https://10.129.0.168:8775/\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:57:24.571993 master-1 kubenswrapper[4771]: I1011 10:57:24.571934 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-combined-ca-bundle\") pod \"ae49cc63-d351-440f-9334-4ef2550565a2\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " Oct 11 10:57:24.572154 master-1 kubenswrapper[4771]: I1011 10:57:24.572017 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-config-data\") pod \"ae49cc63-d351-440f-9334-4ef2550565a2\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " Oct 11 10:57:24.572154 master-1 kubenswrapper[4771]: I1011 10:57:24.572108 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d7dq\" (UniqueName: \"kubernetes.io/projected/ae49cc63-d351-440f-9334-4ef2550565a2-kube-api-access-6d7dq\") pod \"ae49cc63-d351-440f-9334-4ef2550565a2\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " Oct 11 10:57:24.572154 master-1 kubenswrapper[4771]: I1011 10:57:24.572156 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-scripts\") pod \"ae49cc63-d351-440f-9334-4ef2550565a2\" (UID: \"ae49cc63-d351-440f-9334-4ef2550565a2\") " Oct 11 10:57:24.577079 master-1 kubenswrapper[4771]: I1011 10:57:24.576938 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-scripts" (OuterVolumeSpecName: "scripts") pod "ae49cc63-d351-440f-9334-4ef2550565a2" (UID: "ae49cc63-d351-440f-9334-4ef2550565a2"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:24.579472 master-1 kubenswrapper[4771]: I1011 10:57:24.578744 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae49cc63-d351-440f-9334-4ef2550565a2-kube-api-access-6d7dq" (OuterVolumeSpecName: "kube-api-access-6d7dq") pod "ae49cc63-d351-440f-9334-4ef2550565a2" (UID: "ae49cc63-d351-440f-9334-4ef2550565a2"). InnerVolumeSpecName "kube-api-access-6d7dq". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:24.609826 master-1 kubenswrapper[4771]: I1011 10:57:24.599216 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "ae49cc63-d351-440f-9334-4ef2550565a2" (UID: "ae49cc63-d351-440f-9334-4ef2550565a2"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:24.609826 master-1 kubenswrapper[4771]: I1011 10:57:24.600134 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-config-data" (OuterVolumeSpecName: "config-data") pod "ae49cc63-d351-440f-9334-4ef2550565a2" (UID: "ae49cc63-d351-440f-9334-4ef2550565a2"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:24.676801 master-1 kubenswrapper[4771]: I1011 10:57:24.676723 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:24.676801 master-1 kubenswrapper[4771]: I1011 10:57:24.676802 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:24.677080 master-1 kubenswrapper[4771]: I1011 10:57:24.676833 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-6d7dq\" (UniqueName: \"kubernetes.io/projected/ae49cc63-d351-440f-9334-4ef2550565a2-kube-api-access-6d7dq\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:24.677080 master-1 kubenswrapper[4771]: I1011 10:57:24.676857 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/ae49cc63-d351-440f-9334-4ef2550565a2-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:25.030110 master-1 kubenswrapper[4771]: I1011 10:57:25.030038 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerStarted","Data":"dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957"} Oct 11 10:57:25.033677 master-1 kubenswrapper[4771]: I1011 10:57:25.033596 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-host-discover-5f556" event={"ID":"ae49cc63-d351-440f-9334-4ef2550565a2","Type":"ContainerDied","Data":"bf7cd872d077a706534449020eb602438e4892d62ec0c9d5010cb1330868aeea"} Oct 11 10:57:25.033677 master-1 kubenswrapper[4771]: I1011 10:57:25.033665 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf7cd872d077a706534449020eb602438e4892d62ec0c9d5010cb1330868aeea" Oct 11 10:57:25.033677 master-1 kubenswrapper[4771]: I1011 10:57:25.033626 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-host-discover-5f556" Oct 11 10:57:25.037215 master-1 kubenswrapper[4771]: I1011 10:57:25.037130 4771 generic.go:334] "Generic (PLEG): container finished" podID="709c362a-6ace-46bf-9f94-86852f78f6f2" containerID="99d58d9d6b8b62fa18ae8ba7508466dad2a9761e505b9274423ecba095a9de64" exitCode=0 Oct 11 10:57:25.037215 master-1 kubenswrapper[4771]: I1011 10:57:25.037179 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-cell-mapping-bwgtz" event={"ID":"709c362a-6ace-46bf-9f94-86852f78f6f2","Type":"ContainerDied","Data":"99d58d9d6b8b62fa18ae8ba7508466dad2a9761e505b9274423ecba095a9de64"} Oct 11 10:57:26.051022 master-1 kubenswrapper[4771]: I1011 10:57:26.050823 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerStarted","Data":"a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58"} Oct 11 10:57:26.051022 master-1 kubenswrapper[4771]: I1011 10:57:26.050969 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerStarted","Data":"4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78"} Oct 11 10:57:26.631260 master-1 kubenswrapper[4771]: I1011 10:57:26.631203 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:26.735822 master-1 kubenswrapper[4771]: I1011 10:57:26.730167 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-scripts\") pod \"709c362a-6ace-46bf-9f94-86852f78f6f2\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " Oct 11 10:57:26.735822 master-1 kubenswrapper[4771]: I1011 10:57:26.730257 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glspf\" (UniqueName: \"kubernetes.io/projected/709c362a-6ace-46bf-9f94-86852f78f6f2-kube-api-access-glspf\") pod \"709c362a-6ace-46bf-9f94-86852f78f6f2\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " Oct 11 10:57:26.735822 master-1 kubenswrapper[4771]: I1011 10:57:26.730614 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-config-data\") pod \"709c362a-6ace-46bf-9f94-86852f78f6f2\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " Oct 11 10:57:26.735822 master-1 kubenswrapper[4771]: I1011 10:57:26.730812 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-combined-ca-bundle\") pod \"709c362a-6ace-46bf-9f94-86852f78f6f2\" (UID: \"709c362a-6ace-46bf-9f94-86852f78f6f2\") " Oct 11 10:57:26.736343 master-1 kubenswrapper[4771]: I1011 10:57:26.736069 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-scripts" (OuterVolumeSpecName: "scripts") pod "709c362a-6ace-46bf-9f94-86852f78f6f2" (UID: "709c362a-6ace-46bf-9f94-86852f78f6f2"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:26.749459 master-1 kubenswrapper[4771]: I1011 10:57:26.749273 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709c362a-6ace-46bf-9f94-86852f78f6f2-kube-api-access-glspf" (OuterVolumeSpecName: "kube-api-access-glspf") pod "709c362a-6ace-46bf-9f94-86852f78f6f2" (UID: "709c362a-6ace-46bf-9f94-86852f78f6f2"). InnerVolumeSpecName "kube-api-access-glspf". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:26.771495 master-1 kubenswrapper[4771]: I1011 10:57:26.771340 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-config-data" (OuterVolumeSpecName: "config-data") pod "709c362a-6ace-46bf-9f94-86852f78f6f2" (UID: "709c362a-6ace-46bf-9f94-86852f78f6f2"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:26.774741 master-1 kubenswrapper[4771]: I1011 10:57:26.774506 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "709c362a-6ace-46bf-9f94-86852f78f6f2" (UID: "709c362a-6ace-46bf-9f94-86852f78f6f2"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:26.834032 master-1 kubenswrapper[4771]: I1011 10:57:26.833937 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:26.834032 master-1 kubenswrapper[4771]: I1011 10:57:26.833979 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-glspf\" (UniqueName: \"kubernetes.io/projected/709c362a-6ace-46bf-9f94-86852f78f6f2-kube-api-access-glspf\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:26.834032 master-1 kubenswrapper[4771]: I1011 10:57:26.833990 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:26.834032 master-1 kubenswrapper[4771]: I1011 10:57:26.833999 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/709c362a-6ace-46bf-9f94-86852f78f6f2-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:27.067857 master-1 kubenswrapper[4771]: I1011 10:57:27.067776 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-cell1-cell-mapping-bwgtz" event={"ID":"709c362a-6ace-46bf-9f94-86852f78f6f2","Type":"ContainerDied","Data":"4cefc06e3826c53b2bddfc65675ffb15401ad9ff18e58b5e2736c262f90fe5e5"} Oct 11 10:57:27.067857 master-1 kubenswrapper[4771]: I1011 10:57:27.067847 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cefc06e3826c53b2bddfc65675ffb15401ad9ff18e58b5e2736c262f90fe5e5" Oct 11 10:57:27.068849 master-1 kubenswrapper[4771]: I1011 10:57:27.067865 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-cell1-cell-mapping-bwgtz" Oct 11 10:57:28.079261 master-1 kubenswrapper[4771]: I1011 10:57:28.079202 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerStarted","Data":"ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40"} Oct 11 10:57:28.080445 master-1 kubenswrapper[4771]: I1011 10:57:28.080413 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 10:57:28.106951 master-1 kubenswrapper[4771]: I1011 10:57:28.106880 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=2.328467384 podStartE2EDuration="6.106859452s" podCreationTimestamp="2025-10-11 10:57:22 +0000 UTC" firstStartedPulling="2025-10-11 10:57:23.136187533 +0000 UTC m=+1875.110413974" lastFinishedPulling="2025-10-11 10:57:26.914579571 +0000 UTC m=+1878.888806042" observedRunningTime="2025-10-11 10:57:28.10544979 +0000 UTC m=+1880.079676281" watchObservedRunningTime="2025-10-11 10:57:28.106859452 +0000 UTC m=+1880.081085893" Oct 11 10:57:31.323830 master-1 kubenswrapper[4771]: I1011 10:57:31.323707 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 10:57:33.553940 master-1 kubenswrapper[4771]: I1011 10:57:33.553880 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-metadata-1" Oct 11 10:57:33.554748 master-1 kubenswrapper[4771]: I1011 10:57:33.554008 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-metadata-1" Oct 11 10:57:33.561146 master-1 kubenswrapper[4771]: I1011 10:57:33.561047 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-metadata-1" Oct 11 10:57:33.562084 master-1 kubenswrapper[4771]: I1011 10:57:33.562030 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-metadata-1" Oct 11 10:57:42.453382 master-1 kubenswrapper[4771]: I1011 10:57:42.453264 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:42.455621 master-1 kubenswrapper[4771]: I1011 10:57:42.453556 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/nova-scheduler-1" podUID="82101e85-023a-4398-bb5e-4162dea69f46" containerName="nova-scheduler-scheduler" containerID="cri-o://ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc" gracePeriod=30 Oct 11 10:57:43.438780 master-1 kubenswrapper[4771]: E1011 10:57:43.438722 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc" cmd=["/usr/bin/pgrep","-r","DRST","nova-scheduler"] Oct 11 10:57:43.441262 master-1 kubenswrapper[4771]: E1011 10:57:43.441178 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc" cmd=["/usr/bin/pgrep","-r","DRST","nova-scheduler"] Oct 11 10:57:43.443167 master-1 kubenswrapper[4771]: E1011 10:57:43.443133 4771 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc" cmd=["/usr/bin/pgrep","-r","DRST","nova-scheduler"] Oct 11 10:57:43.443439 master-1 kubenswrapper[4771]: E1011 10:57:43.443411 4771 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openstack/nova-scheduler-1" podUID="82101e85-023a-4398-bb5e-4162dea69f46" containerName="nova-scheduler-scheduler" Oct 11 10:57:48.234411 master-1 kubenswrapper[4771]: I1011 10:57:48.234334 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:57:48.312209 master-1 kubenswrapper[4771]: I1011 10:57:48.312110 4771 generic.go:334] "Generic (PLEG): container finished" podID="82101e85-023a-4398-bb5e-4162dea69f46" containerID="ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc" exitCode=0 Oct 11 10:57:48.312209 master-1 kubenswrapper[4771]: I1011 10:57:48.312177 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"82101e85-023a-4398-bb5e-4162dea69f46","Type":"ContainerDied","Data":"ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc"} Oct 11 10:57:48.312569 master-1 kubenswrapper[4771]: I1011 10:57:48.312244 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:57:48.312569 master-1 kubenswrapper[4771]: I1011 10:57:48.312254 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"82101e85-023a-4398-bb5e-4162dea69f46","Type":"ContainerDied","Data":"344c84a1c3165ebb54446b68471bee6244a2b1504eca4e8fa46ae99da6e9b301"} Oct 11 10:57:48.312569 master-1 kubenswrapper[4771]: I1011 10:57:48.312269 4771 scope.go:117] "RemoveContainer" containerID="ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc" Oct 11 10:57:48.340663 master-1 kubenswrapper[4771]: I1011 10:57:48.340562 4771 scope.go:117] "RemoveContainer" containerID="ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc" Oct 11 10:57:48.341338 master-1 kubenswrapper[4771]: E1011 10:57:48.341280 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc\": container with ID starting with ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc not found: ID does not exist" containerID="ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc" Oct 11 10:57:48.341526 master-1 kubenswrapper[4771]: I1011 10:57:48.341486 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc"} err="failed to get container status \"ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc\": rpc error: code = NotFound desc = could not find container \"ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc\": container with ID starting with ceb4ba3a78d59ca572872a25024937ab120ba7e7f85e67747ea7ab0ace9a66fc not found: ID does not exist" Oct 11 10:57:48.410984 master-1 kubenswrapper[4771]: I1011 10:57:48.410721 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh5m8\" (UniqueName: \"kubernetes.io/projected/82101e85-023a-4398-bb5e-4162dea69f46-kube-api-access-kh5m8\") pod \"82101e85-023a-4398-bb5e-4162dea69f46\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " Oct 11 10:57:48.410984 master-1 kubenswrapper[4771]: I1011 10:57:48.410843 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-config-data\") pod \"82101e85-023a-4398-bb5e-4162dea69f46\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " Oct 11 10:57:48.410984 master-1 kubenswrapper[4771]: I1011 10:57:48.410941 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-combined-ca-bundle\") pod \"82101e85-023a-4398-bb5e-4162dea69f46\" (UID: \"82101e85-023a-4398-bb5e-4162dea69f46\") " Oct 11 10:57:48.417343 master-1 kubenswrapper[4771]: I1011 10:57:48.417280 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82101e85-023a-4398-bb5e-4162dea69f46-kube-api-access-kh5m8" (OuterVolumeSpecName: "kube-api-access-kh5m8") pod "82101e85-023a-4398-bb5e-4162dea69f46" (UID: "82101e85-023a-4398-bb5e-4162dea69f46"). InnerVolumeSpecName "kube-api-access-kh5m8". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:48.450702 master-1 kubenswrapper[4771]: I1011 10:57:48.450223 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-config-data" (OuterVolumeSpecName: "config-data") pod "82101e85-023a-4398-bb5e-4162dea69f46" (UID: "82101e85-023a-4398-bb5e-4162dea69f46"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:48.453888 master-1 kubenswrapper[4771]: I1011 10:57:48.453813 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "82101e85-023a-4398-bb5e-4162dea69f46" (UID: "82101e85-023a-4398-bb5e-4162dea69f46"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:48.515567 master-1 kubenswrapper[4771]: I1011 10:57:48.515410 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-kh5m8\" (UniqueName: \"kubernetes.io/projected/82101e85-023a-4398-bb5e-4162dea69f46-kube-api-access-kh5m8\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:48.515567 master-1 kubenswrapper[4771]: I1011 10:57:48.515476 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:48.515567 master-1 kubenswrapper[4771]: I1011 10:57:48.515496 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/82101e85-023a-4398-bb5e-4162dea69f46-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:48.657584 master-1 kubenswrapper[4771]: I1011 10:57:48.656882 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:48.667644 master-1 kubenswrapper[4771]: I1011 10:57:48.667460 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:48.693539 master-1 kubenswrapper[4771]: I1011 10:57:48.693428 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:48.693890 master-1 kubenswrapper[4771]: E1011 10:57:48.693849 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="82101e85-023a-4398-bb5e-4162dea69f46" containerName="nova-scheduler-scheduler" Oct 11 10:57:48.693890 master-1 kubenswrapper[4771]: I1011 10:57:48.693879 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="82101e85-023a-4398-bb5e-4162dea69f46" containerName="nova-scheduler-scheduler" Oct 11 10:57:48.694047 master-1 kubenswrapper[4771]: E1011 10:57:48.693894 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="709c362a-6ace-46bf-9f94-86852f78f6f2" containerName="nova-manage" Oct 11 10:57:48.694047 master-1 kubenswrapper[4771]: I1011 10:57:48.693906 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="709c362a-6ace-46bf-9f94-86852f78f6f2" containerName="nova-manage" Oct 11 10:57:48.694047 master-1 kubenswrapper[4771]: E1011 10:57:48.693957 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="ae49cc63-d351-440f-9334-4ef2550565a2" containerName="nova-manage" Oct 11 10:57:48.694047 master-1 kubenswrapper[4771]: I1011 10:57:48.693966 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae49cc63-d351-440f-9334-4ef2550565a2" containerName="nova-manage" Oct 11 10:57:48.694303 master-1 kubenswrapper[4771]: I1011 10:57:48.694150 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="82101e85-023a-4398-bb5e-4162dea69f46" containerName="nova-scheduler-scheduler" Oct 11 10:57:48.694303 master-1 kubenswrapper[4771]: I1011 10:57:48.694171 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="ae49cc63-d351-440f-9334-4ef2550565a2" containerName="nova-manage" Oct 11 10:57:48.694303 master-1 kubenswrapper[4771]: I1011 10:57:48.694188 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="709c362a-6ace-46bf-9f94-86852f78f6f2" containerName="nova-manage" Oct 11 10:57:48.695013 master-1 kubenswrapper[4771]: I1011 10:57:48.694967 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:57:48.700251 master-1 kubenswrapper[4771]: I1011 10:57:48.700119 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-scheduler-config-data" Oct 11 10:57:48.745764 master-1 kubenswrapper[4771]: I1011 10:57:48.714261 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:48.823470 master-1 kubenswrapper[4771]: I1011 10:57:48.822945 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c-config-data\") pod \"nova-scheduler-1\" (UID: \"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:48.823470 master-1 kubenswrapper[4771]: I1011 10:57:48.823067 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmx7j\" (UniqueName: \"kubernetes.io/projected/77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c-kube-api-access-kmx7j\") pod \"nova-scheduler-1\" (UID: \"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:48.823470 master-1 kubenswrapper[4771]: I1011 10:57:48.823142 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c-combined-ca-bundle\") pod \"nova-scheduler-1\" (UID: \"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:48.926417 master-1 kubenswrapper[4771]: I1011 10:57:48.926184 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c-config-data\") pod \"nova-scheduler-1\" (UID: \"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:48.926417 master-1 kubenswrapper[4771]: I1011 10:57:48.926300 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-kmx7j\" (UniqueName: \"kubernetes.io/projected/77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c-kube-api-access-kmx7j\") pod \"nova-scheduler-1\" (UID: \"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:48.926417 master-1 kubenswrapper[4771]: I1011 10:57:48.926376 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c-combined-ca-bundle\") pod \"nova-scheduler-1\" (UID: \"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:48.931044 master-1 kubenswrapper[4771]: I1011 10:57:48.931000 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c-combined-ca-bundle\") pod \"nova-scheduler-1\" (UID: \"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:48.933503 master-1 kubenswrapper[4771]: I1011 10:57:48.933468 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c-config-data\") pod \"nova-scheduler-1\" (UID: \"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:48.959507 master-1 kubenswrapper[4771]: I1011 10:57:48.959317 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmx7j\" (UniqueName: \"kubernetes.io/projected/77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c-kube-api-access-kmx7j\") pod \"nova-scheduler-1\" (UID: \"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c\") " pod="openstack/nova-scheduler-1" Oct 11 10:57:49.058858 master-1 kubenswrapper[4771]: I1011 10:57:49.058763 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-scheduler-1" Oct 11 10:57:49.534617 master-1 kubenswrapper[4771]: I1011 10:57:49.534543 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-scheduler-1"] Oct 11 10:57:49.535326 master-1 kubenswrapper[4771]: W1011 10:57:49.534639 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77c00ce9_c5ba_47fe_a5c5_98ea5f96a36c.slice/crio-7ac397c3e61ae52c0aeda69f07c9f076ee92106f951040b6ba8a1bb32f43ce21 WatchSource:0}: Error finding container 7ac397c3e61ae52c0aeda69f07c9f076ee92106f951040b6ba8a1bb32f43ce21: Status 404 returned error can't find the container with id 7ac397c3e61ae52c0aeda69f07c9f076ee92106f951040b6ba8a1bb32f43ce21 Oct 11 10:57:50.336186 master-1 kubenswrapper[4771]: I1011 10:57:50.336102 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c","Type":"ContainerStarted","Data":"e7e5c8e5af7f1480b4c2c49099a0be8cb889cec0c16e033ef90c330b388c24bd"} Oct 11 10:57:50.336186 master-1 kubenswrapper[4771]: I1011 10:57:50.336174 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-scheduler-1" event={"ID":"77c00ce9-c5ba-47fe-a5c5-98ea5f96a36c","Type":"ContainerStarted","Data":"7ac397c3e61ae52c0aeda69f07c9f076ee92106f951040b6ba8a1bb32f43ce21"} Oct 11 10:57:50.371973 master-1 kubenswrapper[4771]: I1011 10:57:50.371864 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-scheduler-1" podStartSLOduration=2.371833475 podStartE2EDuration="2.371833475s" podCreationTimestamp="2025-10-11 10:57:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:57:50.363762743 +0000 UTC m=+1902.337989234" watchObservedRunningTime="2025-10-11 10:57:50.371833475 +0000 UTC m=+1902.346059956" Oct 11 10:57:50.464205 master-1 kubenswrapper[4771]: I1011 10:57:50.464094 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82101e85-023a-4398-bb5e-4162dea69f46" path="/var/lib/kubelet/pods/82101e85-023a-4398-bb5e-4162dea69f46/volumes" Oct 11 10:57:52.097296 master-1 kubenswrapper[4771]: I1011 10:57:52.070000 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:52.097296 master-1 kubenswrapper[4771]: I1011 10:57:52.070838 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/nova-metadata-1" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-log" containerID="cri-o://06a1bb6aa8f352b55c65006ce50a6d509e8dff8e484bf4f522e69ff0c42ae932" gracePeriod=30 Oct 11 10:57:52.097296 master-1 kubenswrapper[4771]: I1011 10:57:52.071340 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/nova-metadata-1" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-metadata" containerID="cri-o://80f0dd7f7bf3c1c4412d1e7a39722e2cd092b7c0f670f349af91c500d917aa10" gracePeriod=30 Oct 11 10:57:52.356931 master-1 kubenswrapper[4771]: I1011 10:57:52.356794 4771 generic.go:334] "Generic (PLEG): container finished" podID="486db0a3-f081-43d5-b20d-d7386531632e" containerID="06a1bb6aa8f352b55c65006ce50a6d509e8dff8e484bf4f522e69ff0c42ae932" exitCode=143 Oct 11 10:57:52.356931 master-1 kubenswrapper[4771]: I1011 10:57:52.356858 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"486db0a3-f081-43d5-b20d-d7386531632e","Type":"ContainerDied","Data":"06a1bb6aa8f352b55c65006ce50a6d509e8dff8e484bf4f522e69ff0c42ae932"} Oct 11 10:57:52.590939 master-1 kubenswrapper[4771]: I1011 10:57:52.590846 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ceilometer-0" Oct 11 10:57:54.059826 master-1 kubenswrapper[4771]: I1011 10:57:54.059716 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-scheduler-1" Oct 11 10:57:55.218526 master-1 kubenswrapper[4771]: I1011 10:57:55.218306 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/nova-metadata-1" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-log" probeResult="failure" output="Get \"https://10.129.0.168:8775/\": read tcp 10.129.0.2:47572->10.129.0.168:8775: read: connection reset by peer" Oct 11 10:57:55.220062 master-1 kubenswrapper[4771]: I1011 10:57:55.218481 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/nova-metadata-1" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-metadata" probeResult="failure" output="Get \"https://10.129.0.168:8775/\": read tcp 10.129.0.2:47570->10.129.0.168:8775: read: connection reset by peer" Oct 11 10:57:55.414825 master-1 kubenswrapper[4771]: I1011 10:57:55.414768 4771 generic.go:334] "Generic (PLEG): container finished" podID="486db0a3-f081-43d5-b20d-d7386531632e" containerID="80f0dd7f7bf3c1c4412d1e7a39722e2cd092b7c0f670f349af91c500d917aa10" exitCode=0 Oct 11 10:57:55.414943 master-1 kubenswrapper[4771]: I1011 10:57:55.414834 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"486db0a3-f081-43d5-b20d-d7386531632e","Type":"ContainerDied","Data":"80f0dd7f7bf3c1c4412d1e7a39722e2cd092b7c0f670f349af91c500d917aa10"} Oct 11 10:57:55.750656 master-1 kubenswrapper[4771]: I1011 10:57:55.750586 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:57:55.799248 master-1 kubenswrapper[4771]: I1011 10:57:55.799161 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-config-data\") pod \"486db0a3-f081-43d5-b20d-d7386531632e\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " Oct 11 10:57:55.799662 master-1 kubenswrapper[4771]: I1011 10:57:55.799411 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-combined-ca-bundle\") pod \"486db0a3-f081-43d5-b20d-d7386531632e\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " Oct 11 10:57:55.800086 master-1 kubenswrapper[4771]: I1011 10:57:55.800049 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbq4h\" (UniqueName: \"kubernetes.io/projected/486db0a3-f081-43d5-b20d-d7386531632e-kube-api-access-lbq4h\") pod \"486db0a3-f081-43d5-b20d-d7386531632e\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " Oct 11 10:57:55.800190 master-1 kubenswrapper[4771]: I1011 10:57:55.800113 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"nova-metadata-tls-certs\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-nova-metadata-tls-certs\") pod \"486db0a3-f081-43d5-b20d-d7386531632e\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " Oct 11 10:57:55.800190 master-1 kubenswrapper[4771]: I1011 10:57:55.800173 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/486db0a3-f081-43d5-b20d-d7386531632e-logs\") pod \"486db0a3-f081-43d5-b20d-d7386531632e\" (UID: \"486db0a3-f081-43d5-b20d-d7386531632e\") " Oct 11 10:57:55.801804 master-1 kubenswrapper[4771]: I1011 10:57:55.801708 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486db0a3-f081-43d5-b20d-d7386531632e-logs" (OuterVolumeSpecName: "logs") pod "486db0a3-f081-43d5-b20d-d7386531632e" (UID: "486db0a3-f081-43d5-b20d-d7386531632e"). InnerVolumeSpecName "logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:57:55.832605 master-1 kubenswrapper[4771]: I1011 10:57:55.832489 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486db0a3-f081-43d5-b20d-d7386531632e-kube-api-access-lbq4h" (OuterVolumeSpecName: "kube-api-access-lbq4h") pod "486db0a3-f081-43d5-b20d-d7386531632e" (UID: "486db0a3-f081-43d5-b20d-d7386531632e"). InnerVolumeSpecName "kube-api-access-lbq4h". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:57:55.838788 master-1 kubenswrapper[4771]: I1011 10:57:55.838701 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "486db0a3-f081-43d5-b20d-d7386531632e" (UID: "486db0a3-f081-43d5-b20d-d7386531632e"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:55.848976 master-1 kubenswrapper[4771]: I1011 10:57:55.848619 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-config-data" (OuterVolumeSpecName: "config-data") pod "486db0a3-f081-43d5-b20d-d7386531632e" (UID: "486db0a3-f081-43d5-b20d-d7386531632e"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:55.869578 master-1 kubenswrapper[4771]: I1011 10:57:55.869265 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-nova-metadata-tls-certs" (OuterVolumeSpecName: "nova-metadata-tls-certs") pod "486db0a3-f081-43d5-b20d-d7386531632e" (UID: "486db0a3-f081-43d5-b20d-d7386531632e"). InnerVolumeSpecName "nova-metadata-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:57:55.903922 master-1 kubenswrapper[4771]: I1011 10:57:55.903816 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:55.903922 master-1 kubenswrapper[4771]: I1011 10:57:55.903918 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:55.904213 master-1 kubenswrapper[4771]: I1011 10:57:55.903947 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-lbq4h\" (UniqueName: \"kubernetes.io/projected/486db0a3-f081-43d5-b20d-d7386531632e-kube-api-access-lbq4h\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:55.904213 master-1 kubenswrapper[4771]: I1011 10:57:55.903977 4771 reconciler_common.go:293] "Volume detached for volume \"nova-metadata-tls-certs\" (UniqueName: \"kubernetes.io/secret/486db0a3-f081-43d5-b20d-d7386531632e-nova-metadata-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:55.904213 master-1 kubenswrapper[4771]: I1011 10:57:55.904002 4771 reconciler_common.go:293] "Volume detached for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/486db0a3-f081-43d5-b20d-d7386531632e-logs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:57:56.433280 master-1 kubenswrapper[4771]: I1011 10:57:56.433199 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"486db0a3-f081-43d5-b20d-d7386531632e","Type":"ContainerDied","Data":"5409d9254a9ac28cc3c943a6262472ba07dc81056b84eeb207f5cc4057ceaafd"} Oct 11 10:57:56.434128 master-1 kubenswrapper[4771]: I1011 10:57:56.433299 4771 scope.go:117] "RemoveContainer" containerID="80f0dd7f7bf3c1c4412d1e7a39722e2cd092b7c0f670f349af91c500d917aa10" Oct 11 10:57:56.434128 master-1 kubenswrapper[4771]: I1011 10:57:56.433328 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:57:56.462565 master-1 kubenswrapper[4771]: I1011 10:57:56.462513 4771 scope.go:117] "RemoveContainer" containerID="06a1bb6aa8f352b55c65006ce50a6d509e8dff8e484bf4f522e69ff0c42ae932" Oct 11 10:57:56.490873 master-1 kubenswrapper[4771]: I1011 10:57:56.490725 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:56.503887 master-1 kubenswrapper[4771]: I1011 10:57:56.501610 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:56.524673 master-1 kubenswrapper[4771]: I1011 10:57:56.524585 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:56.525119 master-1 kubenswrapper[4771]: E1011 10:57:56.525076 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-log" Oct 11 10:57:56.525119 master-1 kubenswrapper[4771]: I1011 10:57:56.525100 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-log" Oct 11 10:57:56.525256 master-1 kubenswrapper[4771]: E1011 10:57:56.525131 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-metadata" Oct 11 10:57:56.525256 master-1 kubenswrapper[4771]: I1011 10:57:56.525140 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-metadata" Oct 11 10:57:56.525347 master-1 kubenswrapper[4771]: I1011 10:57:56.525303 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-metadata" Oct 11 10:57:56.525347 master-1 kubenswrapper[4771]: I1011 10:57:56.525316 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="486db0a3-f081-43d5-b20d-d7386531632e" containerName="nova-metadata-log" Oct 11 10:57:56.526907 master-1 kubenswrapper[4771]: I1011 10:57:56.526863 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:57:56.532408 master-1 kubenswrapper[4771]: I1011 10:57:56.532298 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-nova-metadata-internal-svc" Oct 11 10:57:56.532802 master-1 kubenswrapper[4771]: I1011 10:57:56.532715 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-metadata-config-data" Oct 11 10:57:56.550451 master-1 kubenswrapper[4771]: I1011 10:57:56.541519 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:56.620683 master-1 kubenswrapper[4771]: I1011 10:57:56.620474 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-config-data\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.620683 master-1 kubenswrapper[4771]: I1011 10:57:56.620614 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-combined-ca-bundle\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.620683 master-1 kubenswrapper[4771]: I1011 10:57:56.620655 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hhfr\" (UniqueName: \"kubernetes.io/projected/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-kube-api-access-2hhfr\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.621109 master-1 kubenswrapper[4771]: I1011 10:57:56.620763 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-logs\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.621109 master-1 kubenswrapper[4771]: I1011 10:57:56.621028 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-metadata-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-nova-metadata-tls-certs\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.726026 master-1 kubenswrapper[4771]: I1011 10:57:56.725928 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-combined-ca-bundle\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.726530 master-1 kubenswrapper[4771]: I1011 10:57:56.726054 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhfr\" (UniqueName: \"kubernetes.io/projected/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-kube-api-access-2hhfr\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.726530 master-1 kubenswrapper[4771]: I1011 10:57:56.726241 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-logs\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.726530 master-1 kubenswrapper[4771]: I1011 10:57:56.726346 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-metadata-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-nova-metadata-tls-certs\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.726530 master-1 kubenswrapper[4771]: I1011 10:57:56.726456 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-config-data\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.726982 master-1 kubenswrapper[4771]: I1011 10:57:56.726921 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-logs\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.731700 master-1 kubenswrapper[4771]: I1011 10:57:56.731652 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-metadata-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-nova-metadata-tls-certs\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.732092 master-1 kubenswrapper[4771]: I1011 10:57:56.732036 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-config-data\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.732419 master-1 kubenswrapper[4771]: I1011 10:57:56.732391 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-combined-ca-bundle\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.759697 master-1 kubenswrapper[4771]: I1011 10:57:56.759633 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhfr\" (UniqueName: \"kubernetes.io/projected/cc53b2cb-c9f9-46f7-b783-d0a94b4f8060-kube-api-access-2hhfr\") pod \"nova-metadata-1\" (UID: \"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060\") " pod="openstack/nova-metadata-1" Oct 11 10:57:56.862453 master-1 kubenswrapper[4771]: I1011 10:57:56.862330 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-metadata-1" Oct 11 10:57:57.424628 master-1 kubenswrapper[4771]: I1011 10:57:57.424557 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-metadata-1"] Oct 11 10:57:57.445379 master-1 kubenswrapper[4771]: I1011 10:57:57.445281 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060","Type":"ContainerStarted","Data":"56f34d307a3a2b199477fac214ca75a8f159662bd8fe82533b7d7937f345fe83"} Oct 11 10:57:58.451761 master-1 kubenswrapper[4771]: I1011 10:57:58.451129 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486db0a3-f081-43d5-b20d-d7386531632e" path="/var/lib/kubelet/pods/486db0a3-f081-43d5-b20d-d7386531632e/volumes" Oct 11 10:57:58.461877 master-1 kubenswrapper[4771]: I1011 10:57:58.461798 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060","Type":"ContainerStarted","Data":"9a27343f439811da6981cbde217403c4702c9a8eab93426f295b8db491f89051"} Oct 11 10:57:58.461996 master-1 kubenswrapper[4771]: I1011 10:57:58.461884 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-metadata-1" event={"ID":"cc53b2cb-c9f9-46f7-b783-d0a94b4f8060","Type":"ContainerStarted","Data":"7a6d8d4017921ea2d1bb3bd110f46ada1d5f51112971f5f291521d318b2e9588"} Oct 11 10:57:58.500488 master-1 kubenswrapper[4771]: I1011 10:57:58.500380 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-metadata-1" podStartSLOduration=2.500324884 podStartE2EDuration="2.500324884s" podCreationTimestamp="2025-10-11 10:57:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:57:58.490686957 +0000 UTC m=+1910.464913498" watchObservedRunningTime="2025-10-11 10:57:58.500324884 +0000 UTC m=+1910.474551365" Oct 11 10:57:59.059866 master-1 kubenswrapper[4771]: I1011 10:57:59.059771 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-scheduler-1" Oct 11 10:57:59.108602 master-1 kubenswrapper[4771]: I1011 10:57:59.108496 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-scheduler-1" Oct 11 10:57:59.521246 master-1 kubenswrapper[4771]: I1011 10:57:59.521042 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-scheduler-1" Oct 11 10:58:01.862710 master-1 kubenswrapper[4771]: I1011 10:58:01.862641 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-metadata-1" Oct 11 10:58:01.862710 master-1 kubenswrapper[4771]: I1011 10:58:01.862711 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-metadata-1" Oct 11 10:58:06.863094 master-1 kubenswrapper[4771]: I1011 10:58:06.863009 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-metadata-1" Oct 11 10:58:06.863094 master-1 kubenswrapper[4771]: I1011 10:58:06.863091 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-metadata-1" Oct 11 10:58:07.877717 master-1 kubenswrapper[4771]: I1011 10:58:07.877618 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-metadata-1" podUID="cc53b2cb-c9f9-46f7-b783-d0a94b4f8060" containerName="nova-metadata-metadata" probeResult="failure" output="Get \"https://10.129.0.175:8775/\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:58:07.878545 master-1 kubenswrapper[4771]: I1011 10:58:07.877624 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-metadata-1" podUID="cc53b2cb-c9f9-46f7-b783-d0a94b4f8060" containerName="nova-metadata-log" probeResult="failure" output="Get \"https://10.129.0.175:8775/\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:58:15.899072 master-1 kubenswrapper[4771]: I1011 10:58:15.898989 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-api-0"] Oct 11 10:58:15.900295 master-1 kubenswrapper[4771]: I1011 10:58:15.899288 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/nova-api-0" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-log" containerID="cri-o://88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6" gracePeriod=30 Oct 11 10:58:15.900295 master-1 kubenswrapper[4771]: I1011 10:58:15.899890 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/nova-api-0" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-api" containerID="cri-o://684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557" gracePeriod=30 Oct 11 10:58:16.645084 master-1 kubenswrapper[4771]: I1011 10:58:16.644976 4771 generic.go:334] "Generic (PLEG): container finished" podID="c648855c-73f8-4316-9eca-147821b776c2" containerID="88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6" exitCode=143 Oct 11 10:58:16.645084 master-1 kubenswrapper[4771]: I1011 10:58:16.645049 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-0" event={"ID":"c648855c-73f8-4316-9eca-147821b776c2","Type":"ContainerDied","Data":"88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6"} Oct 11 10:58:16.870624 master-1 kubenswrapper[4771]: I1011 10:58:16.870172 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-metadata-1" Oct 11 10:58:16.871032 master-1 kubenswrapper[4771]: I1011 10:58:16.870968 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-metadata-1" Oct 11 10:58:16.881050 master-1 kubenswrapper[4771]: I1011 10:58:16.881001 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-metadata-1" Oct 11 10:58:17.664070 master-1 kubenswrapper[4771]: I1011 10:58:17.663944 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-metadata-1" Oct 11 10:58:19.633740 master-1 kubenswrapper[4771]: I1011 10:58:19.633674 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-0" Oct 11 10:58:19.688828 master-1 kubenswrapper[4771]: I1011 10:58:19.688704 4771 generic.go:334] "Generic (PLEG): container finished" podID="c648855c-73f8-4316-9eca-147821b776c2" containerID="684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557" exitCode=0 Oct 11 10:58:19.688828 master-1 kubenswrapper[4771]: I1011 10:58:19.688770 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-0" Oct 11 10:58:19.689171 master-1 kubenswrapper[4771]: I1011 10:58:19.688807 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-0" event={"ID":"c648855c-73f8-4316-9eca-147821b776c2","Type":"ContainerDied","Data":"684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557"} Oct 11 10:58:19.689171 master-1 kubenswrapper[4771]: I1011 10:58:19.688920 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-0" event={"ID":"c648855c-73f8-4316-9eca-147821b776c2","Type":"ContainerDied","Data":"901bf1754d9ccab6be8ecc339468f500f7f6d434ca5a6c15ca5caad9d817a352"} Oct 11 10:58:19.689171 master-1 kubenswrapper[4771]: I1011 10:58:19.688948 4771 scope.go:117] "RemoveContainer" containerID="684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557" Oct 11 10:58:19.715729 master-1 kubenswrapper[4771]: I1011 10:58:19.715555 4771 scope.go:117] "RemoveContainer" containerID="88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6" Oct 11 10:58:19.735306 master-1 kubenswrapper[4771]: I1011 10:58:19.735246 4771 scope.go:117] "RemoveContainer" containerID="684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557" Oct 11 10:58:19.735808 master-1 kubenswrapper[4771]: E1011 10:58:19.735773 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557\": container with ID starting with 684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557 not found: ID does not exist" containerID="684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557" Oct 11 10:58:19.735931 master-1 kubenswrapper[4771]: I1011 10:58:19.735905 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557"} err="failed to get container status \"684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557\": rpc error: code = NotFound desc = could not find container \"684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557\": container with ID starting with 684566c2bec8831ad596ea2c67283e963a33a72653ccd7d10aa766e1b9786557 not found: ID does not exist" Oct 11 10:58:19.736018 master-1 kubenswrapper[4771]: I1011 10:58:19.736006 4771 scope.go:117] "RemoveContainer" containerID="88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6" Oct 11 10:58:19.736488 master-1 kubenswrapper[4771]: E1011 10:58:19.736340 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6\": container with ID starting with 88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6 not found: ID does not exist" containerID="88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6" Oct 11 10:58:19.736624 master-1 kubenswrapper[4771]: I1011 10:58:19.736606 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6"} err="failed to get container status \"88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6\": rpc error: code = NotFound desc = could not find container \"88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6\": container with ID starting with 88924f324b5ca6c196af8b9bc4aa24510c91c3566f0f572201b83024bd6210b6 not found: ID does not exist" Oct 11 10:58:19.781532 master-1 kubenswrapper[4771]: I1011 10:58:19.781419 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-config-data\") pod \"c648855c-73f8-4316-9eca-147821b776c2\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " Oct 11 10:58:19.781899 master-1 kubenswrapper[4771]: I1011 10:58:19.781565 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-combined-ca-bundle\") pod \"c648855c-73f8-4316-9eca-147821b776c2\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " Oct 11 10:58:19.781899 master-1 kubenswrapper[4771]: I1011 10:58:19.781649 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/c648855c-73f8-4316-9eca-147821b776c2-logs\") pod \"c648855c-73f8-4316-9eca-147821b776c2\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " Oct 11 10:58:19.781899 master-1 kubenswrapper[4771]: I1011 10:58:19.781723 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkcdx\" (UniqueName: \"kubernetes.io/projected/c648855c-73f8-4316-9eca-147821b776c2-kube-api-access-rkcdx\") pod \"c648855c-73f8-4316-9eca-147821b776c2\" (UID: \"c648855c-73f8-4316-9eca-147821b776c2\") " Oct 11 10:58:19.783136 master-1 kubenswrapper[4771]: I1011 10:58:19.783055 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c648855c-73f8-4316-9eca-147821b776c2-logs" (OuterVolumeSpecName: "logs") pod "c648855c-73f8-4316-9eca-147821b776c2" (UID: "c648855c-73f8-4316-9eca-147821b776c2"). InnerVolumeSpecName "logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 10:58:19.786554 master-1 kubenswrapper[4771]: I1011 10:58:19.786499 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c648855c-73f8-4316-9eca-147821b776c2-kube-api-access-rkcdx" (OuterVolumeSpecName: "kube-api-access-rkcdx") pod "c648855c-73f8-4316-9eca-147821b776c2" (UID: "c648855c-73f8-4316-9eca-147821b776c2"). InnerVolumeSpecName "kube-api-access-rkcdx". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 10:58:19.810760 master-1 kubenswrapper[4771]: I1011 10:58:19.810664 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-config-data" (OuterVolumeSpecName: "config-data") pod "c648855c-73f8-4316-9eca-147821b776c2" (UID: "c648855c-73f8-4316-9eca-147821b776c2"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:58:19.819326 master-1 kubenswrapper[4771]: I1011 10:58:19.819266 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "c648855c-73f8-4316-9eca-147821b776c2" (UID: "c648855c-73f8-4316-9eca-147821b776c2"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 10:58:19.885435 master-1 kubenswrapper[4771]: I1011 10:58:19.885324 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 10:58:19.885435 master-1 kubenswrapper[4771]: I1011 10:58:19.885429 4771 reconciler_common.go:293] "Volume detached for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/c648855c-73f8-4316-9eca-147821b776c2-logs\") on node \"master-1\" DevicePath \"\"" Oct 11 10:58:19.885790 master-1 kubenswrapper[4771]: I1011 10:58:19.885461 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-rkcdx\" (UniqueName: \"kubernetes.io/projected/c648855c-73f8-4316-9eca-147821b776c2-kube-api-access-rkcdx\") on node \"master-1\" DevicePath \"\"" Oct 11 10:58:19.885790 master-1 kubenswrapper[4771]: I1011 10:58:19.885490 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c648855c-73f8-4316-9eca-147821b776c2-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 10:58:20.060953 master-1 kubenswrapper[4771]: I1011 10:58:20.060767 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-api-0"] Oct 11 10:58:20.073009 master-1 kubenswrapper[4771]: I1011 10:58:20.072930 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-api-0"] Oct 11 10:58:20.093749 master-1 kubenswrapper[4771]: I1011 10:58:20.093653 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-api-0"] Oct 11 10:58:20.094103 master-1 kubenswrapper[4771]: E1011 10:58:20.094005 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-log" Oct 11 10:58:20.094103 master-1 kubenswrapper[4771]: I1011 10:58:20.094021 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-log" Oct 11 10:58:20.094103 master-1 kubenswrapper[4771]: E1011 10:58:20.094029 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-api" Oct 11 10:58:20.094103 master-1 kubenswrapper[4771]: I1011 10:58:20.094034 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-api" Oct 11 10:58:20.095168 master-1 kubenswrapper[4771]: I1011 10:58:20.094219 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-log" Oct 11 10:58:20.095168 master-1 kubenswrapper[4771]: I1011 10:58:20.094234 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="c648855c-73f8-4316-9eca-147821b776c2" containerName="nova-api-api" Oct 11 10:58:20.095318 master-1 kubenswrapper[4771]: I1011 10:58:20.095238 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-0" Oct 11 10:58:20.098771 master-1 kubenswrapper[4771]: I1011 10:58:20.098704 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-nova-public-svc" Oct 11 10:58:20.099177 master-1 kubenswrapper[4771]: I1011 10:58:20.099109 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-api-config-data" Oct 11 10:58:20.099271 master-1 kubenswrapper[4771]: I1011 10:58:20.099203 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-nova-internal-svc" Oct 11 10:58:20.118866 master-1 kubenswrapper[4771]: I1011 10:58:20.118800 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-api-0"] Oct 11 10:58:20.296856 master-1 kubenswrapper[4771]: I1011 10:58:20.296775 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-combined-ca-bundle\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.297405 master-1 kubenswrapper[4771]: I1011 10:58:20.297369 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-config-data\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.298060 master-1 kubenswrapper[4771]: I1011 10:58:20.298001 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-public-tls-certs\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.298400 master-1 kubenswrapper[4771]: I1011 10:58:20.298336 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69zl\" (UniqueName: \"kubernetes.io/projected/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-kube-api-access-n69zl\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.298744 master-1 kubenswrapper[4771]: I1011 10:58:20.298698 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-logs\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.298992 master-1 kubenswrapper[4771]: I1011 10:58:20.298962 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-internal-tls-certs\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.402141 master-1 kubenswrapper[4771]: I1011 10:58:20.401999 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-combined-ca-bundle\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.402422 master-1 kubenswrapper[4771]: I1011 10:58:20.402156 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-config-data\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.402422 master-1 kubenswrapper[4771]: I1011 10:58:20.402283 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-public-tls-certs\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.402422 master-1 kubenswrapper[4771]: I1011 10:58:20.402394 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-n69zl\" (UniqueName: \"kubernetes.io/projected/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-kube-api-access-n69zl\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.402649 master-1 kubenswrapper[4771]: I1011 10:58:20.402460 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-logs\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.402649 master-1 kubenswrapper[4771]: I1011 10:58:20.402492 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-internal-tls-certs\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.403795 master-1 kubenswrapper[4771]: I1011 10:58:20.403715 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"logs\" (UniqueName: \"kubernetes.io/empty-dir/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-logs\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.408520 master-1 kubenswrapper[4771]: I1011 10:58:20.408449 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-public-tls-certs\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.409074 master-1 kubenswrapper[4771]: I1011 10:58:20.409016 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-internal-tls-certs\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.409667 master-1 kubenswrapper[4771]: I1011 10:58:20.409549 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-config-data\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.410249 master-1 kubenswrapper[4771]: I1011 10:58:20.410212 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-combined-ca-bundle\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.440583 master-1 kubenswrapper[4771]: I1011 10:58:20.440503 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69zl\" (UniqueName: \"kubernetes.io/projected/7dfb8a04-f489-4ab7-b3fa-9477b10e2de0-kube-api-access-n69zl\") pod \"nova-api-0\" (UID: \"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0\") " pod="openstack/nova-api-0" Oct 11 10:58:20.457407 master-1 kubenswrapper[4771]: I1011 10:58:20.457291 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c648855c-73f8-4316-9eca-147821b776c2" path="/var/lib/kubelet/pods/c648855c-73f8-4316-9eca-147821b776c2/volumes" Oct 11 10:58:20.723445 master-1 kubenswrapper[4771]: I1011 10:58:20.723228 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-api-0" Oct 11 10:58:21.254983 master-1 kubenswrapper[4771]: I1011 10:58:21.254935 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-api-0"] Oct 11 10:58:21.720433 master-1 kubenswrapper[4771]: I1011 10:58:21.720331 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-0" event={"ID":"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0","Type":"ContainerStarted","Data":"fecef3be564b816399753f7d8d59265dc3887a1eeeb7163392bbdcd3a103b87a"} Oct 11 10:58:21.720433 master-1 kubenswrapper[4771]: I1011 10:58:21.720407 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-0" event={"ID":"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0","Type":"ContainerStarted","Data":"16c28995ac3dcd1be970af48dc8e48b9c7c87a09a6fa757cf56a58da4eb7ad92"} Oct 11 10:58:22.735830 master-1 kubenswrapper[4771]: I1011 10:58:22.735718 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-api-0" event={"ID":"7dfb8a04-f489-4ab7-b3fa-9477b10e2de0","Type":"ContainerStarted","Data":"00f982736e4546898bb19bea284d3e39908bbc699fd22a8b9544393a4050273b"} Oct 11 10:58:22.780804 master-1 kubenswrapper[4771]: I1011 10:58:22.780653 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-api-0" podStartSLOduration=2.780616743 podStartE2EDuration="2.780616743s" podCreationTimestamp="2025-10-11 10:58:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 10:58:22.773303393 +0000 UTC m=+1934.747529894" watchObservedRunningTime="2025-10-11 10:58:22.780616743 +0000 UTC m=+1934.754843214" Oct 11 10:58:30.724402 master-1 kubenswrapper[4771]: I1011 10:58:30.724259 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-api-0" Oct 11 10:58:30.724402 master-1 kubenswrapper[4771]: I1011 10:58:30.724399 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openstack/nova-api-0" Oct 11 10:58:31.742806 master-1 kubenswrapper[4771]: I1011 10:58:31.742658 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-api-0" podUID="7dfb8a04-f489-4ab7-b3fa-9477b10e2de0" containerName="nova-api-api" probeResult="failure" output="Get \"https://10.129.0.176:8774/\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:58:31.742806 master-1 kubenswrapper[4771]: I1011 10:58:31.742808 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openstack/nova-api-0" podUID="7dfb8a04-f489-4ab7-b3fa-9477b10e2de0" containerName="nova-api-log" probeResult="failure" output="Get \"https://10.129.0.176:8774/\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 11 10:58:40.732621 master-1 kubenswrapper[4771]: I1011 10:58:40.732399 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-api-0" Oct 11 10:58:40.733851 master-1 kubenswrapper[4771]: I1011 10:58:40.733066 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-api-0" Oct 11 10:58:40.734904 master-1 kubenswrapper[4771]: I1011 10:58:40.734833 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openstack/nova-api-0" Oct 11 10:58:40.747903 master-1 kubenswrapper[4771]: I1011 10:58:40.747842 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-api-0" Oct 11 10:58:40.926496 master-1 kubenswrapper[4771]: I1011 10:58:40.925682 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/nova-api-0" Oct 11 10:58:40.939981 master-1 kubenswrapper[4771]: I1011 10:58:40.937377 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/nova-api-0" Oct 11 11:00:00.178176 master-1 kubenswrapper[4771]: I1011 11:00:00.178072 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv"] Oct 11 11:00:00.180574 master-1 kubenswrapper[4771]: I1011 11:00:00.180490 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.183304 master-1 kubenswrapper[4771]: I1011 11:00:00.183238 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-operator-lifecycle-manager"/"pprof-cert" Oct 11 11:00:00.183508 master-1 kubenswrapper[4771]: I1011 11:00:00.183459 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-operator-lifecycle-manager"/"collect-profiles-dockercfg-hbjq2" Oct 11 11:00:00.183896 master-1 kubenswrapper[4771]: I1011 11:00:00.183862 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-operator-lifecycle-manager"/"collect-profiles-config" Oct 11 11:00:00.187722 master-1 kubenswrapper[4771]: I1011 11:00:00.187667 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv"] Oct 11 11:00:00.291456 master-1 kubenswrapper[4771]: I1011 11:00:00.291252 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zpc\" (UniqueName: \"kubernetes.io/projected/52522dc6-1667-4ae1-ba84-82963e615ae0-kube-api-access-m8zpc\") pod \"collect-profiles-29336340-jv5mv\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.291456 master-1 kubenswrapper[4771]: I1011 11:00:00.291346 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52522dc6-1667-4ae1-ba84-82963e615ae0-config-volume\") pod \"collect-profiles-29336340-jv5mv\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.291456 master-1 kubenswrapper[4771]: I1011 11:00:00.291474 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/52522dc6-1667-4ae1-ba84-82963e615ae0-secret-volume\") pod \"collect-profiles-29336340-jv5mv\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.393918 master-1 kubenswrapper[4771]: I1011 11:00:00.393855 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/52522dc6-1667-4ae1-ba84-82963e615ae0-secret-volume\") pod \"collect-profiles-29336340-jv5mv\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.394245 master-1 kubenswrapper[4771]: I1011 11:00:00.393974 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zpc\" (UniqueName: \"kubernetes.io/projected/52522dc6-1667-4ae1-ba84-82963e615ae0-kube-api-access-m8zpc\") pod \"collect-profiles-29336340-jv5mv\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.394245 master-1 kubenswrapper[4771]: I1011 11:00:00.394022 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52522dc6-1667-4ae1-ba84-82963e615ae0-config-volume\") pod \"collect-profiles-29336340-jv5mv\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.395505 master-1 kubenswrapper[4771]: I1011 11:00:00.395036 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52522dc6-1667-4ae1-ba84-82963e615ae0-config-volume\") pod \"collect-profiles-29336340-jv5mv\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.405421 master-1 kubenswrapper[4771]: I1011 11:00:00.398804 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/52522dc6-1667-4ae1-ba84-82963e615ae0-secret-volume\") pod \"collect-profiles-29336340-jv5mv\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.432977 master-1 kubenswrapper[4771]: I1011 11:00:00.432845 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zpc\" (UniqueName: \"kubernetes.io/projected/52522dc6-1667-4ae1-ba84-82963e615ae0-kube-api-access-m8zpc\") pod \"collect-profiles-29336340-jv5mv\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:00.528980 master-1 kubenswrapper[4771]: I1011 11:00:00.528561 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:01.049329 master-1 kubenswrapper[4771]: I1011 11:00:01.049253 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv"] Oct 11 11:00:01.827712 master-1 kubenswrapper[4771]: I1011 11:00:01.827606 4771 generic.go:334] "Generic (PLEG): container finished" podID="52522dc6-1667-4ae1-ba84-82963e615ae0" containerID="66ebd1c6bea5f170d205b6caa3ef551df56836492f79d6c8b4e2315482c9617d" exitCode=0 Oct 11 11:00:01.827712 master-1 kubenswrapper[4771]: I1011 11:00:01.827703 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" event={"ID":"52522dc6-1667-4ae1-ba84-82963e615ae0","Type":"ContainerDied","Data":"66ebd1c6bea5f170d205b6caa3ef551df56836492f79d6c8b4e2315482c9617d"} Oct 11 11:00:01.828415 master-1 kubenswrapper[4771]: I1011 11:00:01.827752 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" event={"ID":"52522dc6-1667-4ae1-ba84-82963e615ae0","Type":"ContainerStarted","Data":"4856a08c0ef28ff8db7f8bf2d0a82acc3d664647b30280176ffde22ed48a00b8"} Oct 11 11:00:03.273834 master-1 kubenswrapper[4771]: I1011 11:00:03.273758 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:03.360925 master-1 kubenswrapper[4771]: I1011 11:00:03.360783 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/52522dc6-1667-4ae1-ba84-82963e615ae0-secret-volume\") pod \"52522dc6-1667-4ae1-ba84-82963e615ae0\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " Oct 11 11:00:03.361233 master-1 kubenswrapper[4771]: I1011 11:00:03.361017 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8zpc\" (UniqueName: \"kubernetes.io/projected/52522dc6-1667-4ae1-ba84-82963e615ae0-kube-api-access-m8zpc\") pod \"52522dc6-1667-4ae1-ba84-82963e615ae0\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " Oct 11 11:00:03.361233 master-1 kubenswrapper[4771]: I1011 11:00:03.361056 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52522dc6-1667-4ae1-ba84-82963e615ae0-config-volume\") pod \"52522dc6-1667-4ae1-ba84-82963e615ae0\" (UID: \"52522dc6-1667-4ae1-ba84-82963e615ae0\") " Oct 11 11:00:03.363727 master-1 kubenswrapper[4771]: I1011 11:00:03.363641 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52522dc6-1667-4ae1-ba84-82963e615ae0-config-volume" (OuterVolumeSpecName: "config-volume") pod "52522dc6-1667-4ae1-ba84-82963e615ae0" (UID: "52522dc6-1667-4ae1-ba84-82963e615ae0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:00:03.364224 master-1 kubenswrapper[4771]: I1011 11:00:03.364025 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52522dc6-1667-4ae1-ba84-82963e615ae0-secret-volume" (OuterVolumeSpecName: "secret-volume") pod "52522dc6-1667-4ae1-ba84-82963e615ae0" (UID: "52522dc6-1667-4ae1-ba84-82963e615ae0"). InnerVolumeSpecName "secret-volume". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:00:03.365533 master-1 kubenswrapper[4771]: I1011 11:00:03.365327 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52522dc6-1667-4ae1-ba84-82963e615ae0-kube-api-access-m8zpc" (OuterVolumeSpecName: "kube-api-access-m8zpc") pod "52522dc6-1667-4ae1-ba84-82963e615ae0" (UID: "52522dc6-1667-4ae1-ba84-82963e615ae0"). InnerVolumeSpecName "kube-api-access-m8zpc". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:00:03.465652 master-1 kubenswrapper[4771]: I1011 11:00:03.465549 4771 reconciler_common.go:293] "Volume detached for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/52522dc6-1667-4ae1-ba84-82963e615ae0-secret-volume\") on node \"master-1\" DevicePath \"\"" Oct 11 11:00:03.465652 master-1 kubenswrapper[4771]: I1011 11:00:03.465591 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-m8zpc\" (UniqueName: \"kubernetes.io/projected/52522dc6-1667-4ae1-ba84-82963e615ae0-kube-api-access-m8zpc\") on node \"master-1\" DevicePath \"\"" Oct 11 11:00:03.465652 master-1 kubenswrapper[4771]: I1011 11:00:03.465604 4771 reconciler_common.go:293] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52522dc6-1667-4ae1-ba84-82963e615ae0-config-volume\") on node \"master-1\" DevicePath \"\"" Oct 11 11:00:03.849761 master-1 kubenswrapper[4771]: I1011 11:00:03.849639 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" event={"ID":"52522dc6-1667-4ae1-ba84-82963e615ae0","Type":"ContainerDied","Data":"4856a08c0ef28ff8db7f8bf2d0a82acc3d664647b30280176ffde22ed48a00b8"} Oct 11 11:00:03.849761 master-1 kubenswrapper[4771]: I1011 11:00:03.849710 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4856a08c0ef28ff8db7f8bf2d0a82acc3d664647b30280176ffde22ed48a00b8" Oct 11 11:00:03.849761 master-1 kubenswrapper[4771]: I1011 11:00:03.849706 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336340-jv5mv" Oct 11 11:00:23.367693 master-1 kubenswrapper[4771]: I1011 11:00:23.367599 4771 scope.go:117] "RemoveContainer" containerID="b9f9706961ea78a9f4e52f8e9ebb80aedc250ae90f55ef49b6cd39d2d53a0f62" Oct 11 11:00:23.412443 master-1 kubenswrapper[4771]: I1011 11:00:23.412348 4771 scope.go:117] "RemoveContainer" containerID="d4634f70346f96ae4f97fe711847f0e072862de8b631ac6a0aaa341026f8675e" Oct 11 11:00:23.480726 master-1 kubenswrapper[4771]: I1011 11:00:23.480683 4771 scope.go:117] "RemoveContainer" containerID="94fe8e005fb0a8b586a5c6a1e344905a51e3390259171a77f131bc97d101f438" Oct 11 11:00:37.893048 master-1 kubenswrapper[4771]: I1011 11:00:37.891643 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-marketplace-ffxw9"] Oct 11 11:00:37.894054 master-1 kubenswrapper[4771]: E1011 11:00:37.893285 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="52522dc6-1667-4ae1-ba84-82963e615ae0" containerName="collect-profiles" Oct 11 11:00:37.894054 master-1 kubenswrapper[4771]: I1011 11:00:37.893309 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="52522dc6-1667-4ae1-ba84-82963e615ae0" containerName="collect-profiles" Oct 11 11:00:37.894054 master-1 kubenswrapper[4771]: I1011 11:00:37.893551 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="52522dc6-1667-4ae1-ba84-82963e615ae0" containerName="collect-profiles" Oct 11 11:00:37.895870 master-1 kubenswrapper[4771]: I1011 11:00:37.895827 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:37.929781 master-1 kubenswrapper[4771]: I1011 11:00:37.929688 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-ffxw9"] Oct 11 11:00:38.016496 master-1 kubenswrapper[4771]: I1011 11:00:38.016416 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-catalog-content\") pod \"redhat-marketplace-ffxw9\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.016783 master-1 kubenswrapper[4771]: I1011 11:00:38.016525 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rgf2\" (UniqueName: \"kubernetes.io/projected/9073372c-472d-49fc-865d-296c5e7e894e-kube-api-access-7rgf2\") pod \"redhat-marketplace-ffxw9\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.016783 master-1 kubenswrapper[4771]: I1011 11:00:38.016611 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-utilities\") pod \"redhat-marketplace-ffxw9\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.118998 master-1 kubenswrapper[4771]: I1011 11:00:38.118474 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-utilities\") pod \"redhat-marketplace-ffxw9\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.118998 master-1 kubenswrapper[4771]: I1011 11:00:38.118598 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-catalog-content\") pod \"redhat-marketplace-ffxw9\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.118998 master-1 kubenswrapper[4771]: I1011 11:00:38.118638 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-7rgf2\" (UniqueName: \"kubernetes.io/projected/9073372c-472d-49fc-865d-296c5e7e894e-kube-api-access-7rgf2\") pod \"redhat-marketplace-ffxw9\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.119388 master-1 kubenswrapper[4771]: I1011 11:00:38.119057 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-utilities\") pod \"redhat-marketplace-ffxw9\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.119475 master-1 kubenswrapper[4771]: I1011 11:00:38.119433 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-catalog-content\") pod \"redhat-marketplace-ffxw9\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.140598 master-1 kubenswrapper[4771]: I1011 11:00:38.140522 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rgf2\" (UniqueName: \"kubernetes.io/projected/9073372c-472d-49fc-865d-296c5e7e894e-kube-api-access-7rgf2\") pod \"redhat-marketplace-ffxw9\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.226857 master-1 kubenswrapper[4771]: I1011 11:00:38.226698 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:38.696130 master-1 kubenswrapper[4771]: I1011 11:00:38.696066 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-ffxw9"] Oct 11 11:00:39.249549 master-1 kubenswrapper[4771]: I1011 11:00:39.249331 4771 generic.go:334] "Generic (PLEG): container finished" podID="9073372c-472d-49fc-865d-296c5e7e894e" containerID="93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2" exitCode=0 Oct 11 11:00:39.249549 master-1 kubenswrapper[4771]: I1011 11:00:39.249417 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-ffxw9" event={"ID":"9073372c-472d-49fc-865d-296c5e7e894e","Type":"ContainerDied","Data":"93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2"} Oct 11 11:00:39.249549 master-1 kubenswrapper[4771]: I1011 11:00:39.249456 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-ffxw9" event={"ID":"9073372c-472d-49fc-865d-296c5e7e894e","Type":"ContainerStarted","Data":"8ea364933c267a126636f7f3d6af92bf9dc8688de7b8628872ff7da70afcc5b6"} Oct 11 11:00:40.264428 master-1 kubenswrapper[4771]: I1011 11:00:40.264287 4771 generic.go:334] "Generic (PLEG): container finished" podID="9073372c-472d-49fc-865d-296c5e7e894e" containerID="c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718" exitCode=0 Oct 11 11:00:40.265411 master-1 kubenswrapper[4771]: I1011 11:00:40.264443 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-ffxw9" event={"ID":"9073372c-472d-49fc-865d-296c5e7e894e","Type":"ContainerDied","Data":"c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718"} Oct 11 11:00:41.277444 master-1 kubenswrapper[4771]: I1011 11:00:41.277328 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-ffxw9" event={"ID":"9073372c-472d-49fc-865d-296c5e7e894e","Type":"ContainerStarted","Data":"b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f"} Oct 11 11:00:41.314851 master-1 kubenswrapper[4771]: I1011 11:00:41.314731 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-marketplace-ffxw9" podStartSLOduration=2.637012764 podStartE2EDuration="4.314704383s" podCreationTimestamp="2025-10-11 11:00:37 +0000 UTC" firstStartedPulling="2025-10-11 11:00:39.252410865 +0000 UTC m=+2071.226637346" lastFinishedPulling="2025-10-11 11:00:40.930102484 +0000 UTC m=+2072.904328965" observedRunningTime="2025-10-11 11:00:41.309824232 +0000 UTC m=+2073.284050673" watchObservedRunningTime="2025-10-11 11:00:41.314704383 +0000 UTC m=+2073.288930864" Oct 11 11:00:48.227479 master-1 kubenswrapper[4771]: I1011 11:00:48.227390 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:48.227479 master-1 kubenswrapper[4771]: I1011 11:00:48.227463 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:48.301311 master-1 kubenswrapper[4771]: I1011 11:00:48.300650 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:48.423241 master-1 kubenswrapper[4771]: I1011 11:00:48.423185 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:48.570862 master-1 kubenswrapper[4771]: I1011 11:00:48.570780 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-ffxw9"] Oct 11 11:00:50.375883 master-1 kubenswrapper[4771]: I1011 11:00:50.375760 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-marketplace-ffxw9" podUID="9073372c-472d-49fc-865d-296c5e7e894e" containerName="registry-server" containerID="cri-o://b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f" gracePeriod=2 Oct 11 11:00:51.018274 master-1 kubenswrapper[4771]: I1011 11:00:51.018182 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:51.175166 master-1 kubenswrapper[4771]: I1011 11:00:51.175054 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rgf2\" (UniqueName: \"kubernetes.io/projected/9073372c-472d-49fc-865d-296c5e7e894e-kube-api-access-7rgf2\") pod \"9073372c-472d-49fc-865d-296c5e7e894e\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " Oct 11 11:00:51.175612 master-1 kubenswrapper[4771]: I1011 11:00:51.175250 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-catalog-content\") pod \"9073372c-472d-49fc-865d-296c5e7e894e\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " Oct 11 11:00:51.175612 master-1 kubenswrapper[4771]: I1011 11:00:51.175293 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-utilities\") pod \"9073372c-472d-49fc-865d-296c5e7e894e\" (UID: \"9073372c-472d-49fc-865d-296c5e7e894e\") " Oct 11 11:00:51.177908 master-1 kubenswrapper[4771]: I1011 11:00:51.177810 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-utilities" (OuterVolumeSpecName: "utilities") pod "9073372c-472d-49fc-865d-296c5e7e894e" (UID: "9073372c-472d-49fc-865d-296c5e7e894e"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:00:51.180713 master-1 kubenswrapper[4771]: I1011 11:00:51.180634 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9073372c-472d-49fc-865d-296c5e7e894e-kube-api-access-7rgf2" (OuterVolumeSpecName: "kube-api-access-7rgf2") pod "9073372c-472d-49fc-865d-296c5e7e894e" (UID: "9073372c-472d-49fc-865d-296c5e7e894e"). InnerVolumeSpecName "kube-api-access-7rgf2". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:00:51.205135 master-1 kubenswrapper[4771]: I1011 11:00:51.204908 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "9073372c-472d-49fc-865d-296c5e7e894e" (UID: "9073372c-472d-49fc-865d-296c5e7e894e"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:00:51.278804 master-1 kubenswrapper[4771]: I1011 11:00:51.278682 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-7rgf2\" (UniqueName: \"kubernetes.io/projected/9073372c-472d-49fc-865d-296c5e7e894e-kube-api-access-7rgf2\") on node \"master-1\" DevicePath \"\"" Oct 11 11:00:51.278804 master-1 kubenswrapper[4771]: I1011 11:00:51.278755 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:00:51.278804 master-1 kubenswrapper[4771]: I1011 11:00:51.278786 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/9073372c-472d-49fc-865d-296c5e7e894e-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:00:51.390596 master-1 kubenswrapper[4771]: I1011 11:00:51.390508 4771 generic.go:334] "Generic (PLEG): container finished" podID="9073372c-472d-49fc-865d-296c5e7e894e" containerID="b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f" exitCode=0 Oct 11 11:00:51.391596 master-1 kubenswrapper[4771]: I1011 11:00:51.390632 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-ffxw9" Oct 11 11:00:51.391596 master-1 kubenswrapper[4771]: I1011 11:00:51.390615 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-ffxw9" event={"ID":"9073372c-472d-49fc-865d-296c5e7e894e","Type":"ContainerDied","Data":"b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f"} Oct 11 11:00:51.391596 master-1 kubenswrapper[4771]: I1011 11:00:51.390879 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-ffxw9" event={"ID":"9073372c-472d-49fc-865d-296c5e7e894e","Type":"ContainerDied","Data":"8ea364933c267a126636f7f3d6af92bf9dc8688de7b8628872ff7da70afcc5b6"} Oct 11 11:00:51.391596 master-1 kubenswrapper[4771]: I1011 11:00:51.390932 4771 scope.go:117] "RemoveContainer" containerID="b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f" Oct 11 11:00:51.416218 master-1 kubenswrapper[4771]: I1011 11:00:51.415659 4771 scope.go:117] "RemoveContainer" containerID="c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718" Oct 11 11:00:51.459900 master-1 kubenswrapper[4771]: I1011 11:00:51.459715 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-ffxw9"] Oct 11 11:00:51.465849 master-1 kubenswrapper[4771]: I1011 11:00:51.465774 4771 scope.go:117] "RemoveContainer" containerID="93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2" Oct 11 11:00:51.467651 master-1 kubenswrapper[4771]: I1011 11:00:51.467590 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-marketplace-ffxw9"] Oct 11 11:00:51.533257 master-1 kubenswrapper[4771]: I1011 11:00:51.533197 4771 scope.go:117] "RemoveContainer" containerID="b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f" Oct 11 11:00:51.534287 master-1 kubenswrapper[4771]: E1011 11:00:51.534200 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f\": container with ID starting with b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f not found: ID does not exist" containerID="b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f" Oct 11 11:00:51.534287 master-1 kubenswrapper[4771]: I1011 11:00:51.534275 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f"} err="failed to get container status \"b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f\": rpc error: code = NotFound desc = could not find container \"b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f\": container with ID starting with b9ffae440e93767bb6dd3cf6faa3e82f54777ec6a2484e2103a6a44c092bc37f not found: ID does not exist" Oct 11 11:00:51.534670 master-1 kubenswrapper[4771]: I1011 11:00:51.534313 4771 scope.go:117] "RemoveContainer" containerID="c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718" Oct 11 11:00:51.535197 master-1 kubenswrapper[4771]: E1011 11:00:51.535136 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718\": container with ID starting with c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718 not found: ID does not exist" containerID="c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718" Oct 11 11:00:51.535197 master-1 kubenswrapper[4771]: I1011 11:00:51.535171 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718"} err="failed to get container status \"c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718\": rpc error: code = NotFound desc = could not find container \"c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718\": container with ID starting with c2bfecdebb12fef33e766ec37e47f3d0cf65379687c38507e2fd4dfbcc98d718 not found: ID does not exist" Oct 11 11:00:51.535197 master-1 kubenswrapper[4771]: I1011 11:00:51.535190 4771 scope.go:117] "RemoveContainer" containerID="93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2" Oct 11 11:00:51.535893 master-1 kubenswrapper[4771]: E1011 11:00:51.535836 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2\": container with ID starting with 93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2 not found: ID does not exist" containerID="93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2" Oct 11 11:00:51.535893 master-1 kubenswrapper[4771]: I1011 11:00:51.535869 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2"} err="failed to get container status \"93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2\": rpc error: code = NotFound desc = could not find container \"93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2\": container with ID starting with 93acab6f93e9fe153fc618c7354e26bfbf7629928b6655f4d192e2bb96b181a2 not found: ID does not exist" Oct 11 11:00:52.456375 master-1 kubenswrapper[4771]: I1011 11:00:52.456250 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9073372c-472d-49fc-865d-296c5e7e894e" path="/var/lib/kubelet/pods/9073372c-472d-49fc-865d-296c5e7e894e/volumes" Oct 11 11:01:00.179169 master-1 kubenswrapper[4771]: I1011 11:01:00.179087 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/keystone-cron-29336341-k4h7v"] Oct 11 11:01:00.182041 master-1 kubenswrapper[4771]: E1011 11:01:00.179497 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="9073372c-472d-49fc-865d-296c5e7e894e" containerName="extract-utilities" Oct 11 11:01:00.182041 master-1 kubenswrapper[4771]: I1011 11:01:00.179515 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="9073372c-472d-49fc-865d-296c5e7e894e" containerName="extract-utilities" Oct 11 11:01:00.182041 master-1 kubenswrapper[4771]: E1011 11:01:00.179550 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="9073372c-472d-49fc-865d-296c5e7e894e" containerName="registry-server" Oct 11 11:01:00.182041 master-1 kubenswrapper[4771]: I1011 11:01:00.179559 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="9073372c-472d-49fc-865d-296c5e7e894e" containerName="registry-server" Oct 11 11:01:00.182041 master-1 kubenswrapper[4771]: E1011 11:01:00.179578 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="9073372c-472d-49fc-865d-296c5e7e894e" containerName="extract-content" Oct 11 11:01:00.182041 master-1 kubenswrapper[4771]: I1011 11:01:00.179587 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="9073372c-472d-49fc-865d-296c5e7e894e" containerName="extract-content" Oct 11 11:01:00.182041 master-1 kubenswrapper[4771]: I1011 11:01:00.179786 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="9073372c-472d-49fc-865d-296c5e7e894e" containerName="registry-server" Oct 11 11:01:00.182041 master-1 kubenswrapper[4771]: I1011 11:01:00.180680 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.203563 master-1 kubenswrapper[4771]: I1011 11:01:00.201836 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/keystone-cron-29336341-k4h7v"] Oct 11 11:01:00.297938 master-1 kubenswrapper[4771]: I1011 11:01:00.297837 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-combined-ca-bundle\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.298296 master-1 kubenswrapper[4771]: I1011 11:01:00.298273 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-config-data\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.298419 master-1 kubenswrapper[4771]: I1011 11:01:00.298385 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rzkz\" (UniqueName: \"kubernetes.io/projected/511b8ba0-8038-431a-8f39-f76a538b45be-kube-api-access-7rzkz\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.298677 master-1 kubenswrapper[4771]: I1011 11:01:00.298607 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"fernet-keys\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-fernet-keys\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.401959 master-1 kubenswrapper[4771]: I1011 11:01:00.401858 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-config-data\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.402281 master-1 kubenswrapper[4771]: I1011 11:01:00.401976 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-7rzkz\" (UniqueName: \"kubernetes.io/projected/511b8ba0-8038-431a-8f39-f76a538b45be-kube-api-access-7rzkz\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.402281 master-1 kubenswrapper[4771]: I1011 11:01:00.402078 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"fernet-keys\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-fernet-keys\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.402281 master-1 kubenswrapper[4771]: I1011 11:01:00.402149 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-combined-ca-bundle\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.408120 master-1 kubenswrapper[4771]: I1011 11:01:00.408003 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-config-data\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.410252 master-1 kubenswrapper[4771]: I1011 11:01:00.410188 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"fernet-keys\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-fernet-keys\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.411602 master-1 kubenswrapper[4771]: I1011 11:01:00.411527 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-combined-ca-bundle\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.424923 master-1 kubenswrapper[4771]: I1011 11:01:00.424849 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rzkz\" (UniqueName: \"kubernetes.io/projected/511b8ba0-8038-431a-8f39-f76a538b45be-kube-api-access-7rzkz\") pod \"keystone-cron-29336341-k4h7v\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:00.507305 master-1 kubenswrapper[4771]: I1011 11:01:00.507090 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:01.047522 master-1 kubenswrapper[4771]: W1011 11:01:01.047238 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod511b8ba0_8038_431a_8f39_f76a538b45be.slice/crio-a3ed1d03b679a3cd94ff44bc23ec01d6de0bca22701ccdc007ac4cf09faceaff WatchSource:0}: Error finding container a3ed1d03b679a3cd94ff44bc23ec01d6de0bca22701ccdc007ac4cf09faceaff: Status 404 returned error can't find the container with id a3ed1d03b679a3cd94ff44bc23ec01d6de0bca22701ccdc007ac4cf09faceaff Oct 11 11:01:01.053614 master-1 kubenswrapper[4771]: I1011 11:01:01.053536 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/keystone-cron-29336341-k4h7v"] Oct 11 11:01:01.502502 master-1 kubenswrapper[4771]: I1011 11:01:01.502342 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/keystone-cron-29336341-k4h7v" event={"ID":"511b8ba0-8038-431a-8f39-f76a538b45be","Type":"ContainerStarted","Data":"8a025688c4f85361b8fd4ce76230c6631cc23efe111a645e578034d939bf0805"} Oct 11 11:01:01.502502 master-1 kubenswrapper[4771]: I1011 11:01:01.502472 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/keystone-cron-29336341-k4h7v" event={"ID":"511b8ba0-8038-431a-8f39-f76a538b45be","Type":"ContainerStarted","Data":"a3ed1d03b679a3cd94ff44bc23ec01d6de0bca22701ccdc007ac4cf09faceaff"} Oct 11 11:01:01.541741 master-1 kubenswrapper[4771]: I1011 11:01:01.541617 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/keystone-cron-29336341-k4h7v" podStartSLOduration=1.541592696 podStartE2EDuration="1.541592696s" podCreationTimestamp="2025-10-11 11:01:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 11:01:01.535678466 +0000 UTC m=+2093.509904907" watchObservedRunningTime="2025-10-11 11:01:01.541592696 +0000 UTC m=+2093.515819157" Oct 11 11:01:03.525672 master-1 kubenswrapper[4771]: I1011 11:01:03.525522 4771 generic.go:334] "Generic (PLEG): container finished" podID="511b8ba0-8038-431a-8f39-f76a538b45be" containerID="8a025688c4f85361b8fd4ce76230c6631cc23efe111a645e578034d939bf0805" exitCode=0 Oct 11 11:01:03.525672 master-1 kubenswrapper[4771]: I1011 11:01:03.525597 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/keystone-cron-29336341-k4h7v" event={"ID":"511b8ba0-8038-431a-8f39-f76a538b45be","Type":"ContainerDied","Data":"8a025688c4f85361b8fd4ce76230c6631cc23efe111a645e578034d939bf0805"} Oct 11 11:01:05.034523 master-1 kubenswrapper[4771]: I1011 11:01:05.034426 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:01:05.049622 master-1 kubenswrapper[4771]: I1011 11:01:05.049404 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"fernet-keys\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-fernet-keys\") pod \"511b8ba0-8038-431a-8f39-f76a538b45be\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " Oct 11 11:01:05.049622 master-1 kubenswrapper[4771]: I1011 11:01:05.049504 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-combined-ca-bundle\") pod \"511b8ba0-8038-431a-8f39-f76a538b45be\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " Oct 11 11:01:05.050083 master-1 kubenswrapper[4771]: I1011 11:01:05.049708 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-config-data\") pod \"511b8ba0-8038-431a-8f39-f76a538b45be\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " Oct 11 11:01:05.050083 master-1 kubenswrapper[4771]: I1011 11:01:05.049759 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rzkz\" (UniqueName: \"kubernetes.io/projected/511b8ba0-8038-431a-8f39-f76a538b45be-kube-api-access-7rzkz\") pod \"511b8ba0-8038-431a-8f39-f76a538b45be\" (UID: \"511b8ba0-8038-431a-8f39-f76a538b45be\") " Oct 11 11:01:05.053410 master-1 kubenswrapper[4771]: I1011 11:01:05.053287 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-fernet-keys" (OuterVolumeSpecName: "fernet-keys") pod "511b8ba0-8038-431a-8f39-f76a538b45be" (UID: "511b8ba0-8038-431a-8f39-f76a538b45be"). InnerVolumeSpecName "fernet-keys". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:01:05.059965 master-1 kubenswrapper[4771]: I1011 11:01:05.059860 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511b8ba0-8038-431a-8f39-f76a538b45be-kube-api-access-7rzkz" (OuterVolumeSpecName: "kube-api-access-7rzkz") pod "511b8ba0-8038-431a-8f39-f76a538b45be" (UID: "511b8ba0-8038-431a-8f39-f76a538b45be"). InnerVolumeSpecName "kube-api-access-7rzkz". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:01:05.098648 master-1 kubenswrapper[4771]: I1011 11:01:05.098568 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "511b8ba0-8038-431a-8f39-f76a538b45be" (UID: "511b8ba0-8038-431a-8f39-f76a538b45be"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:01:05.110442 master-1 kubenswrapper[4771]: I1011 11:01:05.110341 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-config-data" (OuterVolumeSpecName: "config-data") pod "511b8ba0-8038-431a-8f39-f76a538b45be" (UID: "511b8ba0-8038-431a-8f39-f76a538b45be"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:01:05.152574 master-1 kubenswrapper[4771]: I1011 11:01:05.152485 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:01:05.152574 master-1 kubenswrapper[4771]: I1011 11:01:05.152550 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 11:01:05.152574 master-1 kubenswrapper[4771]: I1011 11:01:05.152567 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-7rzkz\" (UniqueName: \"kubernetes.io/projected/511b8ba0-8038-431a-8f39-f76a538b45be-kube-api-access-7rzkz\") on node \"master-1\" DevicePath \"\"" Oct 11 11:01:05.152574 master-1 kubenswrapper[4771]: I1011 11:01:05.152584 4771 reconciler_common.go:293] "Volume detached for volume \"fernet-keys\" (UniqueName: \"kubernetes.io/secret/511b8ba0-8038-431a-8f39-f76a538b45be-fernet-keys\") on node \"master-1\" DevicePath \"\"" Oct 11 11:01:05.549830 master-1 kubenswrapper[4771]: I1011 11:01:05.549727 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/keystone-cron-29336341-k4h7v" event={"ID":"511b8ba0-8038-431a-8f39-f76a538b45be","Type":"ContainerDied","Data":"a3ed1d03b679a3cd94ff44bc23ec01d6de0bca22701ccdc007ac4cf09faceaff"} Oct 11 11:01:05.549830 master-1 kubenswrapper[4771]: I1011 11:01:05.549796 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ed1d03b679a3cd94ff44bc23ec01d6de0bca22701ccdc007ac4cf09faceaff" Oct 11 11:01:05.549830 master-1 kubenswrapper[4771]: I1011 11:01:05.549812 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/keystone-cron-29336341-k4h7v" Oct 11 11:02:11.542298 master-1 kubenswrapper[4771]: I1011 11:02:11.542078 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-db-create-rbtpx"] Oct 11 11:02:11.543182 master-1 kubenswrapper[4771]: E1011 11:02:11.542669 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="511b8ba0-8038-431a-8f39-f76a538b45be" containerName="keystone-cron" Oct 11 11:02:11.543182 master-1 kubenswrapper[4771]: I1011 11:02:11.542691 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="511b8ba0-8038-431a-8f39-f76a538b45be" containerName="keystone-cron" Oct 11 11:02:11.543182 master-1 kubenswrapper[4771]: I1011 11:02:11.542865 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="511b8ba0-8038-431a-8f39-f76a538b45be" containerName="keystone-cron" Oct 11 11:02:11.543876 master-1 kubenswrapper[4771]: I1011 11:02:11.543844 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-db-create-rbtpx" Oct 11 11:02:11.558030 master-1 kubenswrapper[4771]: I1011 11:02:11.557950 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-db-create-rbtpx"] Oct 11 11:02:11.675106 master-1 kubenswrapper[4771]: I1011 11:02:11.675024 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbc8\" (UniqueName: \"kubernetes.io/projected/041373ee-1533-4bc6-abd2-80d16bfa5f23-kube-api-access-srbc8\") pod \"octavia-db-create-rbtpx\" (UID: \"041373ee-1533-4bc6-abd2-80d16bfa5f23\") " pod="openstack/octavia-db-create-rbtpx" Oct 11 11:02:11.778782 master-1 kubenswrapper[4771]: I1011 11:02:11.778683 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-srbc8\" (UniqueName: \"kubernetes.io/projected/041373ee-1533-4bc6-abd2-80d16bfa5f23-kube-api-access-srbc8\") pod \"octavia-db-create-rbtpx\" (UID: \"041373ee-1533-4bc6-abd2-80d16bfa5f23\") " pod="openstack/octavia-db-create-rbtpx" Oct 11 11:02:11.815238 master-1 kubenswrapper[4771]: I1011 11:02:11.814591 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbc8\" (UniqueName: \"kubernetes.io/projected/041373ee-1533-4bc6-abd2-80d16bfa5f23-kube-api-access-srbc8\") pod \"octavia-db-create-rbtpx\" (UID: \"041373ee-1533-4bc6-abd2-80d16bfa5f23\") " pod="openstack/octavia-db-create-rbtpx" Oct 11 11:02:11.875012 master-1 kubenswrapper[4771]: I1011 11:02:11.874911 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-db-create-rbtpx" Oct 11 11:02:12.370168 master-1 kubenswrapper[4771]: I1011 11:02:12.370102 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-db-create-rbtpx"] Oct 11 11:02:13.297111 master-1 kubenswrapper[4771]: I1011 11:02:13.296943 4771 generic.go:334] "Generic (PLEG): container finished" podID="041373ee-1533-4bc6-abd2-80d16bfa5f23" containerID="736c15aefe67b305735f91c4e8c2109ad242f954cfb3635af77a747443410e30" exitCode=0 Oct 11 11:02:13.297111 master-1 kubenswrapper[4771]: I1011 11:02:13.297009 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-db-create-rbtpx" event={"ID":"041373ee-1533-4bc6-abd2-80d16bfa5f23","Type":"ContainerDied","Data":"736c15aefe67b305735f91c4e8c2109ad242f954cfb3635af77a747443410e30"} Oct 11 11:02:13.297111 master-1 kubenswrapper[4771]: I1011 11:02:13.297045 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-db-create-rbtpx" event={"ID":"041373ee-1533-4bc6-abd2-80d16bfa5f23","Type":"ContainerStarted","Data":"59b9f9010aba5846723186241ffea5afeeadfbf4f443023c5bddef0541687b18"} Oct 11 11:02:14.763404 master-1 kubenswrapper[4771]: I1011 11:02:14.763218 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-db-create-rbtpx" Oct 11 11:02:14.797388 master-1 kubenswrapper[4771]: I1011 11:02:14.793319 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srbc8\" (UniqueName: \"kubernetes.io/projected/041373ee-1533-4bc6-abd2-80d16bfa5f23-kube-api-access-srbc8\") pod \"041373ee-1533-4bc6-abd2-80d16bfa5f23\" (UID: \"041373ee-1533-4bc6-abd2-80d16bfa5f23\") " Oct 11 11:02:14.798865 master-1 kubenswrapper[4771]: I1011 11:02:14.798797 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041373ee-1533-4bc6-abd2-80d16bfa5f23-kube-api-access-srbc8" (OuterVolumeSpecName: "kube-api-access-srbc8") pod "041373ee-1533-4bc6-abd2-80d16bfa5f23" (UID: "041373ee-1533-4bc6-abd2-80d16bfa5f23"). InnerVolumeSpecName "kube-api-access-srbc8". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:02:14.895383 master-1 kubenswrapper[4771]: I1011 11:02:14.895280 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-srbc8\" (UniqueName: \"kubernetes.io/projected/041373ee-1533-4bc6-abd2-80d16bfa5f23-kube-api-access-srbc8\") on node \"master-1\" DevicePath \"\"" Oct 11 11:02:15.320832 master-1 kubenswrapper[4771]: I1011 11:02:15.320771 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-db-create-rbtpx" event={"ID":"041373ee-1533-4bc6-abd2-80d16bfa5f23","Type":"ContainerDied","Data":"59b9f9010aba5846723186241ffea5afeeadfbf4f443023c5bddef0541687b18"} Oct 11 11:02:15.321186 master-1 kubenswrapper[4771]: I1011 11:02:15.321164 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59b9f9010aba5846723186241ffea5afeeadfbf4f443023c5bddef0541687b18" Oct 11 11:02:15.321318 master-1 kubenswrapper[4771]: I1011 11:02:15.320813 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-db-create-rbtpx" Oct 11 11:02:23.560495 master-1 kubenswrapper[4771]: I1011 11:02:23.560394 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-e4bf-account-create-m2n6j"] Oct 11 11:02:23.561655 master-1 kubenswrapper[4771]: E1011 11:02:23.561002 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="041373ee-1533-4bc6-abd2-80d16bfa5f23" containerName="mariadb-database-create" Oct 11 11:02:23.561655 master-1 kubenswrapper[4771]: I1011 11:02:23.561024 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="041373ee-1533-4bc6-abd2-80d16bfa5f23" containerName="mariadb-database-create" Oct 11 11:02:23.561655 master-1 kubenswrapper[4771]: I1011 11:02:23.561262 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="041373ee-1533-4bc6-abd2-80d16bfa5f23" containerName="mariadb-database-create" Oct 11 11:02:23.563514 master-1 kubenswrapper[4771]: I1011 11:02:23.562308 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-e4bf-account-create-m2n6j" Oct 11 11:02:23.566755 master-1 kubenswrapper[4771]: I1011 11:02:23.566690 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-db-secret" Oct 11 11:02:23.585438 master-1 kubenswrapper[4771]: I1011 11:02:23.585164 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-e4bf-account-create-m2n6j"] Oct 11 11:02:23.619546 master-1 kubenswrapper[4771]: I1011 11:02:23.619479 4771 scope.go:117] "RemoveContainer" containerID="10573443fa9f81c261e267c2d4f01ad7d7cf7482785a8f4f22c2ccd3fa1fc631" Oct 11 11:02:23.638888 master-1 kubenswrapper[4771]: I1011 11:02:23.638654 4771 scope.go:117] "RemoveContainer" containerID="7f100b006260b4ff812a662ca4646172d63077e3423c5b53974c9a4fc93bb108" Oct 11 11:02:23.657568 master-1 kubenswrapper[4771]: I1011 11:02:23.657510 4771 scope.go:117] "RemoveContainer" containerID="33e1159e64df7103066e5f7850051b2adc3d09e823478d0dc1137ddef2aee326" Oct 11 11:02:23.679407 master-1 kubenswrapper[4771]: I1011 11:02:23.679288 4771 scope.go:117] "RemoveContainer" containerID="1d0d93b3fc6393dcdc851e8c3921d7c5d5a44cf9e99d331f9e66f61b3c48f59d" Oct 11 11:02:23.713379 master-1 kubenswrapper[4771]: I1011 11:02:23.713295 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgsqp\" (UniqueName: \"kubernetes.io/projected/9ba52ba8-24c1-4b0c-83cb-6837e2353fa8-kube-api-access-vgsqp\") pod \"octavia-e4bf-account-create-m2n6j\" (UID: \"9ba52ba8-24c1-4b0c-83cb-6837e2353fa8\") " pod="openstack/octavia-e4bf-account-create-m2n6j" Oct 11 11:02:23.815734 master-1 kubenswrapper[4771]: I1011 11:02:23.815585 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-vgsqp\" (UniqueName: \"kubernetes.io/projected/9ba52ba8-24c1-4b0c-83cb-6837e2353fa8-kube-api-access-vgsqp\") pod \"octavia-e4bf-account-create-m2n6j\" (UID: \"9ba52ba8-24c1-4b0c-83cb-6837e2353fa8\") " pod="openstack/octavia-e4bf-account-create-m2n6j" Oct 11 11:02:23.848951 master-1 kubenswrapper[4771]: I1011 11:02:23.848870 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgsqp\" (UniqueName: \"kubernetes.io/projected/9ba52ba8-24c1-4b0c-83cb-6837e2353fa8-kube-api-access-vgsqp\") pod \"octavia-e4bf-account-create-m2n6j\" (UID: \"9ba52ba8-24c1-4b0c-83cb-6837e2353fa8\") " pod="openstack/octavia-e4bf-account-create-m2n6j" Oct 11 11:02:23.893004 master-1 kubenswrapper[4771]: I1011 11:02:23.892928 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-e4bf-account-create-m2n6j" Oct 11 11:02:24.416924 master-1 kubenswrapper[4771]: I1011 11:02:24.416852 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-e4bf-account-create-m2n6j"] Oct 11 11:02:24.420252 master-1 kubenswrapper[4771]: W1011 11:02:24.420158 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ba52ba8_24c1_4b0c_83cb_6837e2353fa8.slice/crio-0737fbedbdbf7be810d318a8c1d6e9fd5bddd84c6d6ef7682635622648af493e WatchSource:0}: Error finding container 0737fbedbdbf7be810d318a8c1d6e9fd5bddd84c6d6ef7682635622648af493e: Status 404 returned error can't find the container with id 0737fbedbdbf7be810d318a8c1d6e9fd5bddd84c6d6ef7682635622648af493e Oct 11 11:02:25.442696 master-1 kubenswrapper[4771]: I1011 11:02:25.442589 4771 generic.go:334] "Generic (PLEG): container finished" podID="9ba52ba8-24c1-4b0c-83cb-6837e2353fa8" containerID="24ea528e7f6dc70693d8dee3aad4fc9efa2cfed93954344bd9c5720391f051ef" exitCode=0 Oct 11 11:02:25.442696 master-1 kubenswrapper[4771]: I1011 11:02:25.442656 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-e4bf-account-create-m2n6j" event={"ID":"9ba52ba8-24c1-4b0c-83cb-6837e2353fa8","Type":"ContainerDied","Data":"24ea528e7f6dc70693d8dee3aad4fc9efa2cfed93954344bd9c5720391f051ef"} Oct 11 11:02:25.442696 master-1 kubenswrapper[4771]: I1011 11:02:25.442695 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-e4bf-account-create-m2n6j" event={"ID":"9ba52ba8-24c1-4b0c-83cb-6837e2353fa8","Type":"ContainerStarted","Data":"0737fbedbdbf7be810d318a8c1d6e9fd5bddd84c6d6ef7682635622648af493e"} Oct 11 11:02:26.867040 master-1 kubenswrapper[4771]: I1011 11:02:26.866274 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-e4bf-account-create-m2n6j" Oct 11 11:02:26.984282 master-1 kubenswrapper[4771]: I1011 11:02:26.984225 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgsqp\" (UniqueName: \"kubernetes.io/projected/9ba52ba8-24c1-4b0c-83cb-6837e2353fa8-kube-api-access-vgsqp\") pod \"9ba52ba8-24c1-4b0c-83cb-6837e2353fa8\" (UID: \"9ba52ba8-24c1-4b0c-83cb-6837e2353fa8\") " Oct 11 11:02:26.990719 master-1 kubenswrapper[4771]: I1011 11:02:26.990638 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba52ba8-24c1-4b0c-83cb-6837e2353fa8-kube-api-access-vgsqp" (OuterVolumeSpecName: "kube-api-access-vgsqp") pod "9ba52ba8-24c1-4b0c-83cb-6837e2353fa8" (UID: "9ba52ba8-24c1-4b0c-83cb-6837e2353fa8"). InnerVolumeSpecName "kube-api-access-vgsqp". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:02:27.088096 master-1 kubenswrapper[4771]: I1011 11:02:27.088028 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-vgsqp\" (UniqueName: \"kubernetes.io/projected/9ba52ba8-24c1-4b0c-83cb-6837e2353fa8-kube-api-access-vgsqp\") on node \"master-1\" DevicePath \"\"" Oct 11 11:02:27.463928 master-1 kubenswrapper[4771]: I1011 11:02:27.463753 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-e4bf-account-create-m2n6j" event={"ID":"9ba52ba8-24c1-4b0c-83cb-6837e2353fa8","Type":"ContainerDied","Data":"0737fbedbdbf7be810d318a8c1d6e9fd5bddd84c6d6ef7682635622648af493e"} Oct 11 11:02:27.463928 master-1 kubenswrapper[4771]: I1011 11:02:27.463808 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0737fbedbdbf7be810d318a8c1d6e9fd5bddd84c6d6ef7682635622648af493e" Oct 11 11:02:27.463928 master-1 kubenswrapper[4771]: I1011 11:02:27.463906 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-e4bf-account-create-m2n6j" Oct 11 11:02:29.304092 master-1 kubenswrapper[4771]: I1011 11:02:29.303982 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-persistence-db-create-8rff9"] Oct 11 11:02:29.305028 master-1 kubenswrapper[4771]: E1011 11:02:29.304956 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="9ba52ba8-24c1-4b0c-83cb-6837e2353fa8" containerName="mariadb-account-create" Oct 11 11:02:29.305028 master-1 kubenswrapper[4771]: I1011 11:02:29.304979 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba52ba8-24c1-4b0c-83cb-6837e2353fa8" containerName="mariadb-account-create" Oct 11 11:02:29.306108 master-1 kubenswrapper[4771]: I1011 11:02:29.306076 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="9ba52ba8-24c1-4b0c-83cb-6837e2353fa8" containerName="mariadb-account-create" Oct 11 11:02:29.307152 master-1 kubenswrapper[4771]: I1011 11:02:29.307121 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-persistence-db-create-8rff9" Oct 11 11:02:29.320570 master-1 kubenswrapper[4771]: I1011 11:02:29.320492 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-persistence-db-create-8rff9"] Oct 11 11:02:29.443607 master-1 kubenswrapper[4771]: I1011 11:02:29.443511 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscgw\" (UniqueName: \"kubernetes.io/projected/f91fc642-f994-42aa-9bb1-589b5bda7c22-kube-api-access-zscgw\") pod \"octavia-persistence-db-create-8rff9\" (UID: \"f91fc642-f994-42aa-9bb1-589b5bda7c22\") " pod="openstack/octavia-persistence-db-create-8rff9" Oct 11 11:02:29.545929 master-1 kubenswrapper[4771]: I1011 11:02:29.545827 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-zscgw\" (UniqueName: \"kubernetes.io/projected/f91fc642-f994-42aa-9bb1-589b5bda7c22-kube-api-access-zscgw\") pod \"octavia-persistence-db-create-8rff9\" (UID: \"f91fc642-f994-42aa-9bb1-589b5bda7c22\") " pod="openstack/octavia-persistence-db-create-8rff9" Oct 11 11:02:29.572393 master-1 kubenswrapper[4771]: I1011 11:02:29.572220 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscgw\" (UniqueName: \"kubernetes.io/projected/f91fc642-f994-42aa-9bb1-589b5bda7c22-kube-api-access-zscgw\") pod \"octavia-persistence-db-create-8rff9\" (UID: \"f91fc642-f994-42aa-9bb1-589b5bda7c22\") " pod="openstack/octavia-persistence-db-create-8rff9" Oct 11 11:02:29.627383 master-1 kubenswrapper[4771]: I1011 11:02:29.627258 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-persistence-db-create-8rff9" Oct 11 11:02:30.151486 master-1 kubenswrapper[4771]: I1011 11:02:30.151379 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-persistence-db-create-8rff9"] Oct 11 11:02:30.497106 master-1 kubenswrapper[4771]: I1011 11:02:30.497009 4771 generic.go:334] "Generic (PLEG): container finished" podID="f91fc642-f994-42aa-9bb1-589b5bda7c22" containerID="2433d4bbed13285c1fb5cb4b22ca8e93fb7e88d52d9a41f34c5f718dbcf8c96b" exitCode=0 Oct 11 11:02:30.497106 master-1 kubenswrapper[4771]: I1011 11:02:30.497098 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-persistence-db-create-8rff9" event={"ID":"f91fc642-f994-42aa-9bb1-589b5bda7c22","Type":"ContainerDied","Data":"2433d4bbed13285c1fb5cb4b22ca8e93fb7e88d52d9a41f34c5f718dbcf8c96b"} Oct 11 11:02:30.498059 master-1 kubenswrapper[4771]: I1011 11:02:30.497135 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-persistence-db-create-8rff9" event={"ID":"f91fc642-f994-42aa-9bb1-589b5bda7c22","Type":"ContainerStarted","Data":"e627db1dcac557c196093adf0eb1e9981e30cd3c3582197fc5b121e6205873d9"} Oct 11 11:02:31.996077 master-1 kubenswrapper[4771]: I1011 11:02:31.995993 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-persistence-db-create-8rff9" Oct 11 11:02:32.022404 master-1 kubenswrapper[4771]: I1011 11:02:32.022260 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zscgw\" (UniqueName: \"kubernetes.io/projected/f91fc642-f994-42aa-9bb1-589b5bda7c22-kube-api-access-zscgw\") pod \"f91fc642-f994-42aa-9bb1-589b5bda7c22\" (UID: \"f91fc642-f994-42aa-9bb1-589b5bda7c22\") " Oct 11 11:02:32.030548 master-1 kubenswrapper[4771]: I1011 11:02:32.030468 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91fc642-f994-42aa-9bb1-589b5bda7c22-kube-api-access-zscgw" (OuterVolumeSpecName: "kube-api-access-zscgw") pod "f91fc642-f994-42aa-9bb1-589b5bda7c22" (UID: "f91fc642-f994-42aa-9bb1-589b5bda7c22"). InnerVolumeSpecName "kube-api-access-zscgw". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:02:32.125860 master-1 kubenswrapper[4771]: I1011 11:02:32.125752 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-zscgw\" (UniqueName: \"kubernetes.io/projected/f91fc642-f994-42aa-9bb1-589b5bda7c22-kube-api-access-zscgw\") on node \"master-1\" DevicePath \"\"" Oct 11 11:02:32.523336 master-1 kubenswrapper[4771]: I1011 11:02:32.523217 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-persistence-db-create-8rff9" event={"ID":"f91fc642-f994-42aa-9bb1-589b5bda7c22","Type":"ContainerDied","Data":"e627db1dcac557c196093adf0eb1e9981e30cd3c3582197fc5b121e6205873d9"} Oct 11 11:02:32.523336 master-1 kubenswrapper[4771]: I1011 11:02:32.523319 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e627db1dcac557c196093adf0eb1e9981e30cd3c3582197fc5b121e6205873d9" Oct 11 11:02:32.523336 master-1 kubenswrapper[4771]: I1011 11:02:32.523316 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-persistence-db-create-8rff9" Oct 11 11:02:40.244569 master-1 kubenswrapper[4771]: I1011 11:02:40.244467 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-5f0c-account-create-9njpc"] Oct 11 11:02:40.245702 master-1 kubenswrapper[4771]: E1011 11:02:40.245104 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="f91fc642-f994-42aa-9bb1-589b5bda7c22" containerName="mariadb-database-create" Oct 11 11:02:40.245702 master-1 kubenswrapper[4771]: I1011 11:02:40.245140 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91fc642-f994-42aa-9bb1-589b5bda7c22" containerName="mariadb-database-create" Oct 11 11:02:40.245702 master-1 kubenswrapper[4771]: I1011 11:02:40.245577 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="f91fc642-f994-42aa-9bb1-589b5bda7c22" containerName="mariadb-database-create" Oct 11 11:02:40.248686 master-1 kubenswrapper[4771]: I1011 11:02:40.247207 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-5f0c-account-create-9njpc" Oct 11 11:02:40.250996 master-1 kubenswrapper[4771]: I1011 11:02:40.250576 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-persistence-db-secret" Oct 11 11:02:40.293469 master-1 kubenswrapper[4771]: I1011 11:02:40.293400 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-5f0c-account-create-9njpc"] Oct 11 11:02:40.418209 master-1 kubenswrapper[4771]: I1011 11:02:40.417954 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5p2\" (UniqueName: \"kubernetes.io/projected/3eb36428-2086-42ca-8ebf-9864a0917971-kube-api-access-hh5p2\") pod \"octavia-5f0c-account-create-9njpc\" (UID: \"3eb36428-2086-42ca-8ebf-9864a0917971\") " pod="openstack/octavia-5f0c-account-create-9njpc" Oct 11 11:02:40.520921 master-1 kubenswrapper[4771]: I1011 11:02:40.520772 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5p2\" (UniqueName: \"kubernetes.io/projected/3eb36428-2086-42ca-8ebf-9864a0917971-kube-api-access-hh5p2\") pod \"octavia-5f0c-account-create-9njpc\" (UID: \"3eb36428-2086-42ca-8ebf-9864a0917971\") " pod="openstack/octavia-5f0c-account-create-9njpc" Oct 11 11:02:40.557527 master-1 kubenswrapper[4771]: I1011 11:02:40.557456 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5p2\" (UniqueName: \"kubernetes.io/projected/3eb36428-2086-42ca-8ebf-9864a0917971-kube-api-access-hh5p2\") pod \"octavia-5f0c-account-create-9njpc\" (UID: \"3eb36428-2086-42ca-8ebf-9864a0917971\") " pod="openstack/octavia-5f0c-account-create-9njpc" Oct 11 11:02:40.585274 master-1 kubenswrapper[4771]: I1011 11:02:40.585226 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-5f0c-account-create-9njpc" Oct 11 11:02:41.108019 master-1 kubenswrapper[4771]: I1011 11:02:41.107945 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-5f0c-account-create-9njpc"] Oct 11 11:02:41.115799 master-1 kubenswrapper[4771]: W1011 11:02:41.115704 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eb36428_2086_42ca_8ebf_9864a0917971.slice/crio-ae6f1ba6fa658a954ed1dd9fd98150e308b5f8c5446f2f5e0899071d5a4e95fd WatchSource:0}: Error finding container ae6f1ba6fa658a954ed1dd9fd98150e308b5f8c5446f2f5e0899071d5a4e95fd: Status 404 returned error can't find the container with id ae6f1ba6fa658a954ed1dd9fd98150e308b5f8c5446f2f5e0899071d5a4e95fd Oct 11 11:02:41.635956 master-1 kubenswrapper[4771]: I1011 11:02:41.635900 4771 generic.go:334] "Generic (PLEG): container finished" podID="3eb36428-2086-42ca-8ebf-9864a0917971" containerID="9343b079fc4a104c0cb1564885c29bbb8cf2e14829a4f1096f11a9696ef57edf" exitCode=0 Oct 11 11:02:41.635956 master-1 kubenswrapper[4771]: I1011 11:02:41.635949 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-5f0c-account-create-9njpc" event={"ID":"3eb36428-2086-42ca-8ebf-9864a0917971","Type":"ContainerDied","Data":"9343b079fc4a104c0cb1564885c29bbb8cf2e14829a4f1096f11a9696ef57edf"} Oct 11 11:02:41.639404 master-1 kubenswrapper[4771]: I1011 11:02:41.636005 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-5f0c-account-create-9njpc" event={"ID":"3eb36428-2086-42ca-8ebf-9864a0917971","Type":"ContainerStarted","Data":"ae6f1ba6fa658a954ed1dd9fd98150e308b5f8c5446f2f5e0899071d5a4e95fd"} Oct 11 11:02:43.260753 master-1 kubenswrapper[4771]: I1011 11:02:43.260673 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-5f0c-account-create-9njpc" Oct 11 11:02:43.429582 master-1 kubenswrapper[4771]: I1011 11:02:43.429387 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh5p2\" (UniqueName: \"kubernetes.io/projected/3eb36428-2086-42ca-8ebf-9864a0917971-kube-api-access-hh5p2\") pod \"3eb36428-2086-42ca-8ebf-9864a0917971\" (UID: \"3eb36428-2086-42ca-8ebf-9864a0917971\") " Oct 11 11:02:43.434043 master-1 kubenswrapper[4771]: I1011 11:02:43.433932 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb36428-2086-42ca-8ebf-9864a0917971-kube-api-access-hh5p2" (OuterVolumeSpecName: "kube-api-access-hh5p2") pod "3eb36428-2086-42ca-8ebf-9864a0917971" (UID: "3eb36428-2086-42ca-8ebf-9864a0917971"). InnerVolumeSpecName "kube-api-access-hh5p2". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:02:43.533169 master-1 kubenswrapper[4771]: I1011 11:02:43.533039 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-hh5p2\" (UniqueName: \"kubernetes.io/projected/3eb36428-2086-42ca-8ebf-9864a0917971-kube-api-access-hh5p2\") on node \"master-1\" DevicePath \"\"" Oct 11 11:02:43.668323 master-1 kubenswrapper[4771]: I1011 11:02:43.668137 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-5f0c-account-create-9njpc" event={"ID":"3eb36428-2086-42ca-8ebf-9864a0917971","Type":"ContainerDied","Data":"ae6f1ba6fa658a954ed1dd9fd98150e308b5f8c5446f2f5e0899071d5a4e95fd"} Oct 11 11:02:43.668323 master-1 kubenswrapper[4771]: I1011 11:02:43.668203 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae6f1ba6fa658a954ed1dd9fd98150e308b5f8c5446f2f5e0899071d5a4e95fd" Oct 11 11:02:43.668323 master-1 kubenswrapper[4771]: I1011 11:02:43.668235 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-5f0c-account-create-9njpc" Oct 11 11:02:46.257954 master-1 kubenswrapper[4771]: I1011 11:02:46.256757 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-api-84f885c68-ttgvk"] Oct 11 11:02:46.257954 master-1 kubenswrapper[4771]: E1011 11:02:46.257954 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="3eb36428-2086-42ca-8ebf-9864a0917971" containerName="mariadb-account-create" Oct 11 11:02:46.259208 master-1 kubenswrapper[4771]: I1011 11:02:46.257984 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb36428-2086-42ca-8ebf-9864a0917971" containerName="mariadb-account-create" Oct 11 11:02:46.259208 master-1 kubenswrapper[4771]: I1011 11:02:46.258235 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="3eb36428-2086-42ca-8ebf-9864a0917971" containerName="mariadb-account-create" Oct 11 11:02:46.260581 master-1 kubenswrapper[4771]: I1011 11:02:46.260469 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.264094 master-1 kubenswrapper[4771]: I1011 11:02:46.263922 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-api-scripts" Oct 11 11:02:46.264094 master-1 kubenswrapper[4771]: I1011 11:02:46.264050 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-api-config-data" Oct 11 11:02:46.265799 master-1 kubenswrapper[4771]: I1011 11:02:46.265689 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-octavia-ovndbs" Oct 11 11:02:46.274235 master-1 kubenswrapper[4771]: I1011 11:02:46.274164 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-api-84f885c68-ttgvk"] Oct 11 11:02:46.401216 master-1 kubenswrapper[4771]: I1011 11:02:46.401066 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-scripts\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.401757 master-1 kubenswrapper[4771]: I1011 11:02:46.401722 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data-merged\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.401914 master-1 kubenswrapper[4771]: I1011 11:02:46.401895 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"octavia-run\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-octavia-run\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.402216 master-1 kubenswrapper[4771]: I1011 11:02:46.402170 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.402491 master-1 kubenswrapper[4771]: I1011 11:02:46.402435 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-ovndb-tls-certs\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.402602 master-1 kubenswrapper[4771]: I1011 11:02:46.402542 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-combined-ca-bundle\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.504237 master-1 kubenswrapper[4771]: I1011 11:02:46.504016 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.504237 master-1 kubenswrapper[4771]: I1011 11:02:46.504150 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-ovndb-tls-certs\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.504237 master-1 kubenswrapper[4771]: I1011 11:02:46.504204 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-combined-ca-bundle\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.504237 master-1 kubenswrapper[4771]: I1011 11:02:46.504262 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-scripts\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.510729 master-1 kubenswrapper[4771]: I1011 11:02:46.510684 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data-merged\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.510729 master-1 kubenswrapper[4771]: I1011 11:02:46.510733 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"octavia-run\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-octavia-run\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.515310 master-1 kubenswrapper[4771]: I1011 11:02:46.511433 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"octavia-run\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-octavia-run\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.515310 master-1 kubenswrapper[4771]: I1011 11:02:46.511758 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data-merged\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.515310 master-1 kubenswrapper[4771]: I1011 11:02:46.514560 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-ovndb-tls-certs\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.515310 master-1 kubenswrapper[4771]: I1011 11:02:46.514718 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-combined-ca-bundle\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.515310 master-1 kubenswrapper[4771]: I1011 11:02:46.515243 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.527360 master-1 kubenswrapper[4771]: I1011 11:02:46.527282 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-scripts\") pod \"octavia-api-84f885c68-ttgvk\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:46.588543 master-1 kubenswrapper[4771]: I1011 11:02:46.588427 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:47.101993 master-1 kubenswrapper[4771]: I1011 11:02:47.101925 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-api-84f885c68-ttgvk"] Oct 11 11:02:47.108008 master-1 kubenswrapper[4771]: W1011 11:02:47.107751 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda67756a2_aa42_4f6f_b27a_57e16f566883.slice/crio-a5a886249a2d7b53c7a4945ae527e497dd08083a42ca3105272e41c3a3ba5d39 WatchSource:0}: Error finding container a5a886249a2d7b53c7a4945ae527e497dd08083a42ca3105272e41c3a3ba5d39: Status 404 returned error can't find the container with id a5a886249a2d7b53c7a4945ae527e497dd08083a42ca3105272e41c3a3ba5d39 Oct 11 11:02:47.110086 master-1 kubenswrapper[4771]: I1011 11:02:47.110025 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 11:02:47.722872 master-1 kubenswrapper[4771]: I1011 11:02:47.722790 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-84f885c68-ttgvk" event={"ID":"a67756a2-aa42-4f6f-b27a-57e16f566883","Type":"ContainerStarted","Data":"a5a886249a2d7b53c7a4945ae527e497dd08083a42ca3105272e41c3a3ba5d39"} Oct 11 11:02:57.852219 master-1 kubenswrapper[4771]: I1011 11:02:57.852139 4771 generic.go:334] "Generic (PLEG): container finished" podID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerID="36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c" exitCode=0 Oct 11 11:02:57.853352 master-1 kubenswrapper[4771]: I1011 11:02:57.852233 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-84f885c68-ttgvk" event={"ID":"a67756a2-aa42-4f6f-b27a-57e16f566883","Type":"ContainerDied","Data":"36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c"} Oct 11 11:02:58.867821 master-1 kubenswrapper[4771]: I1011 11:02:58.867746 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-84f885c68-ttgvk" event={"ID":"a67756a2-aa42-4f6f-b27a-57e16f566883","Type":"ContainerStarted","Data":"e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598"} Oct 11 11:02:58.867821 master-1 kubenswrapper[4771]: I1011 11:02:58.867829 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-84f885c68-ttgvk" event={"ID":"a67756a2-aa42-4f6f-b27a-57e16f566883","Type":"ContainerStarted","Data":"4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530"} Oct 11 11:02:58.869798 master-1 kubenswrapper[4771]: I1011 11:02:58.869728 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:58.869798 master-1 kubenswrapper[4771]: I1011 11:02:58.869791 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:02:58.906386 master-1 kubenswrapper[4771]: I1011 11:02:58.906217 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/octavia-api-84f885c68-ttgvk" podStartSLOduration=3.061617878 podStartE2EDuration="12.906186922s" podCreationTimestamp="2025-10-11 11:02:46 +0000 UTC" firstStartedPulling="2025-10-11 11:02:47.109582472 +0000 UTC m=+2199.083808923" lastFinishedPulling="2025-10-11 11:02:56.954151516 +0000 UTC m=+2208.928377967" observedRunningTime="2025-10-11 11:02:58.903870966 +0000 UTC m=+2210.878097427" watchObservedRunningTime="2025-10-11 11:02:58.906186922 +0000 UTC m=+2210.880413403" Oct 11 11:03:00.002906 master-1 kubenswrapper[4771]: I1011 11:03:00.002805 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:00.003826 master-1 kubenswrapper[4771]: I1011 11:03:00.003215 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="ceilometer-central-agent" containerID="cri-o://dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957" gracePeriod=30 Oct 11 11:03:00.003826 master-1 kubenswrapper[4771]: I1011 11:03:00.003568 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="ceilometer-notification-agent" containerID="cri-o://4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78" gracePeriod=30 Oct 11 11:03:00.003826 master-1 kubenswrapper[4771]: I1011 11:03:00.003571 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="proxy-httpd" containerID="cri-o://ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40" gracePeriod=30 Oct 11 11:03:00.004846 master-1 kubenswrapper[4771]: I1011 11:03:00.003795 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="sg-core" containerID="cri-o://a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58" gracePeriod=30 Oct 11 11:03:00.891519 master-1 kubenswrapper[4771]: I1011 11:03:00.891445 4771 generic.go:334] "Generic (PLEG): container finished" podID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerID="ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40" exitCode=0 Oct 11 11:03:00.891519 master-1 kubenswrapper[4771]: I1011 11:03:00.891498 4771 generic.go:334] "Generic (PLEG): container finished" podID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerID="a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58" exitCode=2 Oct 11 11:03:00.891519 master-1 kubenswrapper[4771]: I1011 11:03:00.891509 4771 generic.go:334] "Generic (PLEG): container finished" podID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerID="dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957" exitCode=0 Oct 11 11:03:00.891872 master-1 kubenswrapper[4771]: I1011 11:03:00.891524 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerDied","Data":"ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40"} Oct 11 11:03:00.891872 master-1 kubenswrapper[4771]: I1011 11:03:00.891615 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerDied","Data":"a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58"} Oct 11 11:03:00.891872 master-1 kubenswrapper[4771]: I1011 11:03:00.891648 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerDied","Data":"dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957"} Oct 11 11:03:01.613141 master-1 kubenswrapper[4771]: I1011 11:03:01.613086 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.713797 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-combined-ca-bundle\") pod \"926f8cdc-bbf6-4328-8436-8428df0a679b\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.713894 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwhv2\" (UniqueName: \"kubernetes.io/projected/926f8cdc-bbf6-4328-8436-8428df0a679b-kube-api-access-xwhv2\") pod \"926f8cdc-bbf6-4328-8436-8428df0a679b\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.713958 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-sg-core-conf-yaml\") pod \"926f8cdc-bbf6-4328-8436-8428df0a679b\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.713990 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-run-httpd\") pod \"926f8cdc-bbf6-4328-8436-8428df0a679b\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.714012 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-ceilometer-tls-certs\") pod \"926f8cdc-bbf6-4328-8436-8428df0a679b\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.714037 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-log-httpd\") pod \"926f8cdc-bbf6-4328-8436-8428df0a679b\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.714110 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-config-data\") pod \"926f8cdc-bbf6-4328-8436-8428df0a679b\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.714154 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-scripts\") pod \"926f8cdc-bbf6-4328-8436-8428df0a679b\" (UID: \"926f8cdc-bbf6-4328-8436-8428df0a679b\") " Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.715635 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "926f8cdc-bbf6-4328-8436-8428df0a679b" (UID: "926f8cdc-bbf6-4328-8436-8428df0a679b"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.716080 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "926f8cdc-bbf6-4328-8436-8428df0a679b" (UID: "926f8cdc-bbf6-4328-8436-8428df0a679b"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:03:01.719414 master-1 kubenswrapper[4771]: I1011 11:03:01.718152 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-scripts" (OuterVolumeSpecName: "scripts") pod "926f8cdc-bbf6-4328-8436-8428df0a679b" (UID: "926f8cdc-bbf6-4328-8436-8428df0a679b"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:01.721454 master-1 kubenswrapper[4771]: I1011 11:03:01.720793 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926f8cdc-bbf6-4328-8436-8428df0a679b-kube-api-access-xwhv2" (OuterVolumeSpecName: "kube-api-access-xwhv2") pod "926f8cdc-bbf6-4328-8436-8428df0a679b" (UID: "926f8cdc-bbf6-4328-8436-8428df0a679b"). InnerVolumeSpecName "kube-api-access-xwhv2". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:03:01.746615 master-1 kubenswrapper[4771]: I1011 11:03:01.746561 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "926f8cdc-bbf6-4328-8436-8428df0a679b" (UID: "926f8cdc-bbf6-4328-8436-8428df0a679b"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:01.793774 master-1 kubenswrapper[4771]: I1011 11:03:01.793608 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-ceilometer-tls-certs" (OuterVolumeSpecName: "ceilometer-tls-certs") pod "926f8cdc-bbf6-4328-8436-8428df0a679b" (UID: "926f8cdc-bbf6-4328-8436-8428df0a679b"). InnerVolumeSpecName "ceilometer-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:01.797306 master-1 kubenswrapper[4771]: I1011 11:03:01.797230 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "926f8cdc-bbf6-4328-8436-8428df0a679b" (UID: "926f8cdc-bbf6-4328-8436-8428df0a679b"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:01.816524 master-1 kubenswrapper[4771]: I1011 11:03:01.816450 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-xwhv2\" (UniqueName: \"kubernetes.io/projected/926f8cdc-bbf6-4328-8436-8428df0a679b-kube-api-access-xwhv2\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:01.816524 master-1 kubenswrapper[4771]: I1011 11:03:01.816499 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:01.816524 master-1 kubenswrapper[4771]: I1011 11:03:01.816510 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:01.816524 master-1 kubenswrapper[4771]: I1011 11:03:01.816519 4771 reconciler_common.go:293] "Volume detached for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-ceilometer-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:01.816524 master-1 kubenswrapper[4771]: I1011 11:03:01.816529 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/926f8cdc-bbf6-4328-8436-8428df0a679b-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:01.816524 master-1 kubenswrapper[4771]: I1011 11:03:01.816537 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:01.816832 master-1 kubenswrapper[4771]: I1011 11:03:01.816546 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:01.827199 master-1 kubenswrapper[4771]: I1011 11:03:01.827133 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-config-data" (OuterVolumeSpecName: "config-data") pod "926f8cdc-bbf6-4328-8436-8428df0a679b" (UID: "926f8cdc-bbf6-4328-8436-8428df0a679b"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:01.901750 master-1 kubenswrapper[4771]: I1011 11:03:01.901672 4771 generic.go:334] "Generic (PLEG): container finished" podID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerID="4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78" exitCode=0 Oct 11 11:03:01.901750 master-1 kubenswrapper[4771]: I1011 11:03:01.901729 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerDied","Data":"4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78"} Oct 11 11:03:01.901750 master-1 kubenswrapper[4771]: I1011 11:03:01.901756 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"926f8cdc-bbf6-4328-8436-8428df0a679b","Type":"ContainerDied","Data":"2a75cd52ad1f72be5ccb56e9952a02a3c2bfd8c3f845acfbe551f4d25daeffc2"} Oct 11 11:03:01.902082 master-1 kubenswrapper[4771]: I1011 11:03:01.901774 4771 scope.go:117] "RemoveContainer" containerID="ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40" Oct 11 11:03:01.902082 master-1 kubenswrapper[4771]: I1011 11:03:01.901903 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 11:03:01.917872 master-1 kubenswrapper[4771]: I1011 11:03:01.917817 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/926f8cdc-bbf6-4328-8436-8428df0a679b-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:01.931806 master-1 kubenswrapper[4771]: I1011 11:03:01.931701 4771 scope.go:117] "RemoveContainer" containerID="a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58" Oct 11 11:03:01.945032 master-1 kubenswrapper[4771]: I1011 11:03:01.944967 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:01.972625 master-1 kubenswrapper[4771]: I1011 11:03:01.968088 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:01.978462 master-1 kubenswrapper[4771]: I1011 11:03:01.978403 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:01.978724 master-1 kubenswrapper[4771]: E1011 11:03:01.978688 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="ceilometer-central-agent" Oct 11 11:03:01.978724 master-1 kubenswrapper[4771]: I1011 11:03:01.978707 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="ceilometer-central-agent" Oct 11 11:03:01.978724 master-1 kubenswrapper[4771]: E1011 11:03:01.978718 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="ceilometer-notification-agent" Oct 11 11:03:01.978724 master-1 kubenswrapper[4771]: I1011 11:03:01.978724 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="ceilometer-notification-agent" Oct 11 11:03:01.978724 master-1 kubenswrapper[4771]: E1011 11:03:01.978731 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="proxy-httpd" Oct 11 11:03:01.979053 master-1 kubenswrapper[4771]: I1011 11:03:01.978739 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="proxy-httpd" Oct 11 11:03:01.979053 master-1 kubenswrapper[4771]: E1011 11:03:01.978750 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="sg-core" Oct 11 11:03:01.979053 master-1 kubenswrapper[4771]: I1011 11:03:01.978757 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="sg-core" Oct 11 11:03:01.979053 master-1 kubenswrapper[4771]: I1011 11:03:01.978894 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="ceilometer-central-agent" Oct 11 11:03:01.979053 master-1 kubenswrapper[4771]: I1011 11:03:01.978911 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="ceilometer-notification-agent" Oct 11 11:03:01.979053 master-1 kubenswrapper[4771]: I1011 11:03:01.978918 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="sg-core" Oct 11 11:03:01.979053 master-1 kubenswrapper[4771]: I1011 11:03:01.978929 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" containerName="proxy-httpd" Oct 11 11:03:01.980472 master-1 kubenswrapper[4771]: I1011 11:03:01.980330 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 11:03:01.980570 master-1 kubenswrapper[4771]: I1011 11:03:01.980520 4771 scope.go:117] "RemoveContainer" containerID="4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78" Oct 11 11:03:01.985621 master-1 kubenswrapper[4771]: I1011 11:03:01.985580 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 11:03:01.985856 master-1 kubenswrapper[4771]: I1011 11:03:01.985810 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 11:03:01.985856 master-1 kubenswrapper[4771]: I1011 11:03:01.985830 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ceilometer-internal-svc" Oct 11 11:03:01.999793 master-1 kubenswrapper[4771]: I1011 11:03:01.999736 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:02.016958 master-1 kubenswrapper[4771]: I1011 11:03:02.016891 4771 scope.go:117] "RemoveContainer" containerID="dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957" Oct 11 11:03:02.044702 master-1 kubenswrapper[4771]: I1011 11:03:02.044634 4771 scope.go:117] "RemoveContainer" containerID="ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40" Oct 11 11:03:02.045189 master-1 kubenswrapper[4771]: E1011 11:03:02.045132 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40\": container with ID starting with ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40 not found: ID does not exist" containerID="ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40" Oct 11 11:03:02.045189 master-1 kubenswrapper[4771]: I1011 11:03:02.045179 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40"} err="failed to get container status \"ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40\": rpc error: code = NotFound desc = could not find container \"ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40\": container with ID starting with ca31db39d1dec9084865eb81efc92049edf7d6166fd06121934c9ffcdc701d40 not found: ID does not exist" Oct 11 11:03:02.045409 master-1 kubenswrapper[4771]: I1011 11:03:02.045206 4771 scope.go:117] "RemoveContainer" containerID="a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58" Oct 11 11:03:02.045803 master-1 kubenswrapper[4771]: E1011 11:03:02.045739 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58\": container with ID starting with a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58 not found: ID does not exist" containerID="a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58" Oct 11 11:03:02.045803 master-1 kubenswrapper[4771]: I1011 11:03:02.045783 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58"} err="failed to get container status \"a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58\": rpc error: code = NotFound desc = could not find container \"a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58\": container with ID starting with a7cc850a7be7b2d0fcc12f07e1e079159798fbdcd81356efd79d9ef6b76f3e58 not found: ID does not exist" Oct 11 11:03:02.045803 master-1 kubenswrapper[4771]: I1011 11:03:02.045797 4771 scope.go:117] "RemoveContainer" containerID="4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78" Oct 11 11:03:02.046236 master-1 kubenswrapper[4771]: E1011 11:03:02.046177 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78\": container with ID starting with 4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78 not found: ID does not exist" containerID="4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78" Oct 11 11:03:02.046236 master-1 kubenswrapper[4771]: I1011 11:03:02.046218 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78"} err="failed to get container status \"4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78\": rpc error: code = NotFound desc = could not find container \"4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78\": container with ID starting with 4c4740c654b3eb5bc657fa26dafcc70a7dbbb11d03e1c33a59b6b35165113d78 not found: ID does not exist" Oct 11 11:03:02.046467 master-1 kubenswrapper[4771]: I1011 11:03:02.046252 4771 scope.go:117] "RemoveContainer" containerID="dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957" Oct 11 11:03:02.046759 master-1 kubenswrapper[4771]: E1011 11:03:02.046715 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957\": container with ID starting with dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957 not found: ID does not exist" containerID="dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957" Oct 11 11:03:02.046759 master-1 kubenswrapper[4771]: I1011 11:03:02.046744 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957"} err="failed to get container status \"dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957\": rpc error: code = NotFound desc = could not find container \"dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957\": container with ID starting with dae10efe0437b4662fa70f9c02ae261a84610fc2231a80eb15286d70bfdd6957 not found: ID does not exist" Oct 11 11:03:02.122767 master-1 kubenswrapper[4771]: I1011 11:03:02.122581 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-scripts\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.122767 master-1 kubenswrapper[4771]: I1011 11:03:02.122730 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-config-data\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.123252 master-1 kubenswrapper[4771]: I1011 11:03:02.122845 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc67p\" (UniqueName: \"kubernetes.io/projected/d199f8f0-ae54-4c5c-b8af-f3058547edf1-kube-api-access-jc67p\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.123252 master-1 kubenswrapper[4771]: I1011 11:03:02.122926 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-log-httpd\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.123252 master-1 kubenswrapper[4771]: I1011 11:03:02.123005 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-run-httpd\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.123252 master-1 kubenswrapper[4771]: I1011 11:03:02.123063 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.123252 master-1 kubenswrapper[4771]: I1011 11:03:02.123223 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.123647 master-1 kubenswrapper[4771]: I1011 11:03:02.123283 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.225767 master-1 kubenswrapper[4771]: I1011 11:03:02.225617 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-jc67p\" (UniqueName: \"kubernetes.io/projected/d199f8f0-ae54-4c5c-b8af-f3058547edf1-kube-api-access-jc67p\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.226566 master-1 kubenswrapper[4771]: I1011 11:03:02.226521 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-log-httpd\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.226669 master-1 kubenswrapper[4771]: I1011 11:03:02.226613 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-log-httpd\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.226753 master-1 kubenswrapper[4771]: I1011 11:03:02.226738 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-run-httpd\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.226964 master-1 kubenswrapper[4771]: I1011 11:03:02.226909 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.227061 master-1 kubenswrapper[4771]: I1011 11:03:02.227031 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.227131 master-1 kubenswrapper[4771]: I1011 11:03:02.227068 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.227331 master-1 kubenswrapper[4771]: I1011 11:03:02.227280 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-scripts\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.227331 master-1 kubenswrapper[4771]: I1011 11:03:02.227292 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-run-httpd\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.227528 master-1 kubenswrapper[4771]: I1011 11:03:02.227326 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-config-data\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.233855 master-1 kubenswrapper[4771]: I1011 11:03:02.233754 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.236713 master-1 kubenswrapper[4771]: I1011 11:03:02.236659 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-scripts\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.239545 master-1 kubenswrapper[4771]: I1011 11:03:02.239486 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.239820 master-1 kubenswrapper[4771]: I1011 11:03:02.239771 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-config-data\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.249146 master-1 kubenswrapper[4771]: I1011 11:03:02.248948 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.262209 master-1 kubenswrapper[4771]: I1011 11:03:02.262139 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc67p\" (UniqueName: \"kubernetes.io/projected/d199f8f0-ae54-4c5c-b8af-f3058547edf1-kube-api-access-jc67p\") pod \"ceilometer-0\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " pod="openstack/ceilometer-0" Oct 11 11:03:02.300860 master-1 kubenswrapper[4771]: I1011 11:03:02.300775 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 11:03:02.469208 master-1 kubenswrapper[4771]: I1011 11:03:02.469117 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926f8cdc-bbf6-4328-8436-8428df0a679b" path="/var/lib/kubelet/pods/926f8cdc-bbf6-4328-8436-8428df0a679b/volumes" Oct 11 11:03:02.846929 master-1 kubenswrapper[4771]: I1011 11:03:02.846823 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:02.850419 master-1 kubenswrapper[4771]: W1011 11:03:02.850349 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd199f8f0_ae54_4c5c_b8af_f3058547edf1.slice/crio-1f5edb60e620fedfd68b508b4b511f9365534396f20ac698191ef7d2d59c6bfb WatchSource:0}: Error finding container 1f5edb60e620fedfd68b508b4b511f9365534396f20ac698191ef7d2d59c6bfb: Status 404 returned error can't find the container with id 1f5edb60e620fedfd68b508b4b511f9365534396f20ac698191ef7d2d59c6bfb Oct 11 11:03:02.914704 master-1 kubenswrapper[4771]: I1011 11:03:02.914612 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerStarted","Data":"1f5edb60e620fedfd68b508b4b511f9365534396f20ac698191ef7d2d59c6bfb"} Oct 11 11:03:03.605603 master-1 kubenswrapper[4771]: I1011 11:03:03.604762 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-rsyslog-ljmzz"] Oct 11 11:03:03.606546 master-1 kubenswrapper[4771]: I1011 11:03:03.606513 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.610571 master-1 kubenswrapper[4771]: I1011 11:03:03.610516 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-rsyslog-config-data" Oct 11 11:03:03.612186 master-1 kubenswrapper[4771]: I1011 11:03:03.612116 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-rsyslog-scripts" Oct 11 11:03:03.614413 master-1 kubenswrapper[4771]: I1011 11:03:03.613267 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"octavia-hmport-map" Oct 11 11:03:03.620477 master-1 kubenswrapper[4771]: I1011 11:03:03.620182 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-rsyslog-ljmzz"] Oct 11 11:03:03.769882 master-1 kubenswrapper[4771]: I1011 11:03:03.769827 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/e71d1867-29ec-46d8-9057-7b862e658e4c-hm-ports\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.770122 master-1 kubenswrapper[4771]: I1011 11:03:03.769943 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/e71d1867-29ec-46d8-9057-7b862e658e4c-config-data-merged\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.770122 master-1 kubenswrapper[4771]: I1011 11:03:03.769977 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e71d1867-29ec-46d8-9057-7b862e658e4c-config-data\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.770122 master-1 kubenswrapper[4771]: I1011 11:03:03.770034 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e71d1867-29ec-46d8-9057-7b862e658e4c-scripts\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.872309 master-1 kubenswrapper[4771]: I1011 11:03:03.872186 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/e71d1867-29ec-46d8-9057-7b862e658e4c-hm-ports\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.872309 master-1 kubenswrapper[4771]: I1011 11:03:03.872289 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/e71d1867-29ec-46d8-9057-7b862e658e4c-config-data-merged\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.872834 master-1 kubenswrapper[4771]: I1011 11:03:03.872311 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e71d1867-29ec-46d8-9057-7b862e658e4c-config-data\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.872834 master-1 kubenswrapper[4771]: I1011 11:03:03.872349 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e71d1867-29ec-46d8-9057-7b862e658e4c-scripts\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.872901 master-1 kubenswrapper[4771]: I1011 11:03:03.872829 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/e71d1867-29ec-46d8-9057-7b862e658e4c-config-data-merged\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.873943 master-1 kubenswrapper[4771]: I1011 11:03:03.873883 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/e71d1867-29ec-46d8-9057-7b862e658e4c-hm-ports\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.875874 master-1 kubenswrapper[4771]: I1011 11:03:03.875760 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/e71d1867-29ec-46d8-9057-7b862e658e4c-config-data\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.876012 master-1 kubenswrapper[4771]: I1011 11:03:03.875986 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/e71d1867-29ec-46d8-9057-7b862e658e4c-scripts\") pod \"octavia-rsyslog-ljmzz\" (UID: \"e71d1867-29ec-46d8-9057-7b862e658e4c\") " pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:03.924843 master-1 kubenswrapper[4771]: I1011 11:03:03.924778 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerStarted","Data":"2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea"} Oct 11 11:03:03.929780 master-1 kubenswrapper[4771]: I1011 11:03:03.929740 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:04.475904 master-1 kubenswrapper[4771]: I1011 11:03:04.475818 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-image-upload-678599687f-vbxnh"] Oct 11 11:03:04.477741 master-1 kubenswrapper[4771]: I1011 11:03:04.477706 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:04.486450 master-1 kubenswrapper[4771]: I1011 11:03:04.483001 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-config-data" Oct 11 11:03:04.505075 master-1 kubenswrapper[4771]: I1011 11:03:04.504990 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-image-upload-678599687f-vbxnh"] Oct 11 11:03:04.538345 master-1 kubenswrapper[4771]: I1011 11:03:04.537269 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-rsyslog-ljmzz"] Oct 11 11:03:04.599381 master-1 kubenswrapper[4771]: I1011 11:03:04.599265 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/30a9a1fe-e08c-4112-a7b8-6616d280405e-httpd-config\") pod \"octavia-image-upload-678599687f-vbxnh\" (UID: \"30a9a1fe-e08c-4112-a7b8-6616d280405e\") " pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:04.599793 master-1 kubenswrapper[4771]: I1011 11:03:04.599498 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"amphora-image\" (UniqueName: \"kubernetes.io/empty-dir/30a9a1fe-e08c-4112-a7b8-6616d280405e-amphora-image\") pod \"octavia-image-upload-678599687f-vbxnh\" (UID: \"30a9a1fe-e08c-4112-a7b8-6616d280405e\") " pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:04.708037 master-1 kubenswrapper[4771]: I1011 11:03:04.706504 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/30a9a1fe-e08c-4112-a7b8-6616d280405e-httpd-config\") pod \"octavia-image-upload-678599687f-vbxnh\" (UID: \"30a9a1fe-e08c-4112-a7b8-6616d280405e\") " pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:04.708037 master-1 kubenswrapper[4771]: I1011 11:03:04.706741 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"amphora-image\" (UniqueName: \"kubernetes.io/empty-dir/30a9a1fe-e08c-4112-a7b8-6616d280405e-amphora-image\") pod \"octavia-image-upload-678599687f-vbxnh\" (UID: \"30a9a1fe-e08c-4112-a7b8-6616d280405e\") " pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:04.708037 master-1 kubenswrapper[4771]: I1011 11:03:04.707600 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"amphora-image\" (UniqueName: \"kubernetes.io/empty-dir/30a9a1fe-e08c-4112-a7b8-6616d280405e-amphora-image\") pod \"octavia-image-upload-678599687f-vbxnh\" (UID: \"30a9a1fe-e08c-4112-a7b8-6616d280405e\") " pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:04.718005 master-1 kubenswrapper[4771]: I1011 11:03:04.717890 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/30a9a1fe-e08c-4112-a7b8-6616d280405e-httpd-config\") pod \"octavia-image-upload-678599687f-vbxnh\" (UID: \"30a9a1fe-e08c-4112-a7b8-6616d280405e\") " pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:04.804649 master-1 kubenswrapper[4771]: I1011 11:03:04.804606 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:04.940844 master-1 kubenswrapper[4771]: I1011 11:03:04.940780 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerStarted","Data":"f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692"} Oct 11 11:03:04.948768 master-1 kubenswrapper[4771]: I1011 11:03:04.942482 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-rsyslog-ljmzz" event={"ID":"e71d1867-29ec-46d8-9057-7b862e658e4c","Type":"ContainerStarted","Data":"20ff9e10bb2272bee7130877e2cd8a446c12de1b611c03f2f5b4990b7f22c4e1"} Oct 11 11:03:05.207291 master-1 kubenswrapper[4771]: I1011 11:03:05.205411 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-db-sync-dsphh"] Oct 11 11:03:05.207291 master-1 kubenswrapper[4771]: I1011 11:03:05.207203 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.211688 master-1 kubenswrapper[4771]: I1011 11:03:05.211628 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-scripts" Oct 11 11:03:05.221396 master-1 kubenswrapper[4771]: I1011 11:03:05.221315 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-db-sync-dsphh"] Oct 11 11:03:05.289116 master-1 kubenswrapper[4771]: I1011 11:03:05.289031 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-image-upload-678599687f-vbxnh"] Oct 11 11:03:05.294575 master-1 kubenswrapper[4771]: W1011 11:03:05.294499 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30a9a1fe_e08c_4112_a7b8_6616d280405e.slice/crio-822d91bade265914abdf5fc87f453e2f88a3133976d26e99e2272117f587739c WatchSource:0}: Error finding container 822d91bade265914abdf5fc87f453e2f88a3133976d26e99e2272117f587739c: Status 404 returned error can't find the container with id 822d91bade265914abdf5fc87f453e2f88a3133976d26e99e2272117f587739c Oct 11 11:03:05.322876 master-1 kubenswrapper[4771]: I1011 11:03:05.322813 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data-merged\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.322876 master-1 kubenswrapper[4771]: I1011 11:03:05.322885 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-scripts\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.323194 master-1 kubenswrapper[4771]: I1011 11:03:05.323013 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.323194 master-1 kubenswrapper[4771]: I1011 11:03:05.323034 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-combined-ca-bundle\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.427384 master-1 kubenswrapper[4771]: I1011 11:03:05.426376 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.427384 master-1 kubenswrapper[4771]: I1011 11:03:05.426530 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-combined-ca-bundle\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.427384 master-1 kubenswrapper[4771]: I1011 11:03:05.427167 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data-merged\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.427384 master-1 kubenswrapper[4771]: I1011 11:03:05.427278 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-scripts\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.428560 master-1 kubenswrapper[4771]: I1011 11:03:05.428323 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data-merged\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.451483 master-1 kubenswrapper[4771]: I1011 11:03:05.431319 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-combined-ca-bundle\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.451483 master-1 kubenswrapper[4771]: I1011 11:03:05.435145 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-scripts\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.451483 master-1 kubenswrapper[4771]: I1011 11:03:05.437009 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data\") pod \"octavia-db-sync-dsphh\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.534428 master-1 kubenswrapper[4771]: I1011 11:03:05.534340 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:05.972332 master-1 kubenswrapper[4771]: I1011 11:03:05.972260 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerStarted","Data":"ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3"} Oct 11 11:03:05.973780 master-1 kubenswrapper[4771]: I1011 11:03:05.973731 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-image-upload-678599687f-vbxnh" event={"ID":"30a9a1fe-e08c-4112-a7b8-6616d280405e","Type":"ContainerStarted","Data":"822d91bade265914abdf5fc87f453e2f88a3133976d26e99e2272117f587739c"} Oct 11 11:03:06.990280 master-1 kubenswrapper[4771]: I1011 11:03:06.990205 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerStarted","Data":"d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f"} Oct 11 11:03:06.991859 master-1 kubenswrapper[4771]: I1011 11:03:06.991794 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 11:03:07.060854 master-1 kubenswrapper[4771]: I1011 11:03:07.060782 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-db-sync-dsphh"] Oct 11 11:03:07.085219 master-1 kubenswrapper[4771]: I1011 11:03:07.085138 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=2.733738982 podStartE2EDuration="6.085112906s" podCreationTimestamp="2025-10-11 11:03:01 +0000 UTC" firstStartedPulling="2025-10-11 11:03:02.85537011 +0000 UTC m=+2214.829596551" lastFinishedPulling="2025-10-11 11:03:06.206744034 +0000 UTC m=+2218.180970475" observedRunningTime="2025-10-11 11:03:07.057683569 +0000 UTC m=+2219.031910040" watchObservedRunningTime="2025-10-11 11:03:07.085112906 +0000 UTC m=+2219.059339347" Oct 11 11:03:07.666454 master-1 kubenswrapper[4771]: W1011 11:03:07.660622 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2681ddac_5e31_449b_bf71_fb54e8ba389c.slice/crio-62529d885f50e38d46775355d42d0bf6618c28a31161c124af0a288723c5173b WatchSource:0}: Error finding container 62529d885f50e38d46775355d42d0bf6618c28a31161c124af0a288723c5173b: Status 404 returned error can't find the container with id 62529d885f50e38d46775355d42d0bf6618c28a31161c124af0a288723c5173b Oct 11 11:03:08.008258 master-1 kubenswrapper[4771]: I1011 11:03:08.008110 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-db-sync-dsphh" event={"ID":"2681ddac-5e31-449b-bf71-fb54e8ba389c","Type":"ContainerStarted","Data":"62529d885f50e38d46775355d42d0bf6618c28a31161c124af0a288723c5173b"} Oct 11 11:03:09.019432 master-1 kubenswrapper[4771]: I1011 11:03:09.019231 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-rsyslog-ljmzz" event={"ID":"e71d1867-29ec-46d8-9057-7b862e658e4c","Type":"ContainerStarted","Data":"10eedbd9c1cf8319085bd869b6939fe0341c197bd1a92f749d507e8b4b690ca9"} Oct 11 11:03:09.023722 master-1 kubenswrapper[4771]: I1011 11:03:09.023665 4771 generic.go:334] "Generic (PLEG): container finished" podID="2681ddac-5e31-449b-bf71-fb54e8ba389c" containerID="649f9407421e9587f48e63d8c49281e9025d233acd0bf83983ed52b03d671758" exitCode=0 Oct 11 11:03:09.023875 master-1 kubenswrapper[4771]: I1011 11:03:09.023714 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-db-sync-dsphh" event={"ID":"2681ddac-5e31-449b-bf71-fb54e8ba389c","Type":"ContainerDied","Data":"649f9407421e9587f48e63d8c49281e9025d233acd0bf83983ed52b03d671758"} Oct 11 11:03:10.033576 master-1 kubenswrapper[4771]: I1011 11:03:10.033333 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-db-sync-dsphh" event={"ID":"2681ddac-5e31-449b-bf71-fb54e8ba389c","Type":"ContainerStarted","Data":"00d99e2ba51100c415ae5d1a3b19ce4ab68cd0b4655796bec9fd8f7ec75f10f8"} Oct 11 11:03:10.223286 master-1 kubenswrapper[4771]: I1011 11:03:10.223191 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/octavia-db-sync-dsphh" podStartSLOduration=5.22316825 podStartE2EDuration="5.22316825s" podCreationTimestamp="2025-10-11 11:03:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 11:03:10.120954408 +0000 UTC m=+2222.095180869" watchObservedRunningTime="2025-10-11 11:03:10.22316825 +0000 UTC m=+2222.197394711" Oct 11 11:03:11.045965 master-1 kubenswrapper[4771]: I1011 11:03:11.045866 4771 generic.go:334] "Generic (PLEG): container finished" podID="e71d1867-29ec-46d8-9057-7b862e658e4c" containerID="10eedbd9c1cf8319085bd869b6939fe0341c197bd1a92f749d507e8b4b690ca9" exitCode=0 Oct 11 11:03:11.047768 master-1 kubenswrapper[4771]: I1011 11:03:11.047711 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-rsyslog-ljmzz" event={"ID":"e71d1867-29ec-46d8-9057-7b862e658e4c","Type":"ContainerDied","Data":"10eedbd9c1cf8319085bd869b6939fe0341c197bd1a92f749d507e8b4b690ca9"} Oct 11 11:03:14.095433 master-1 kubenswrapper[4771]: I1011 11:03:14.095278 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-rsyslog-ljmzz" event={"ID":"e71d1867-29ec-46d8-9057-7b862e658e4c","Type":"ContainerStarted","Data":"ab76b9ef3a27720c3ed7bbf5d3d905989fa800df9a8b2ba7b35a422c01cf124c"} Oct 11 11:03:14.095982 master-1 kubenswrapper[4771]: I1011 11:03:14.095668 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:14.123040 master-1 kubenswrapper[4771]: I1011 11:03:14.122939 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/octavia-rsyslog-ljmzz" podStartSLOduration=2.659351729 podStartE2EDuration="11.122916259s" podCreationTimestamp="2025-10-11 11:03:03 +0000 UTC" firstStartedPulling="2025-10-11 11:03:04.541572819 +0000 UTC m=+2216.515799260" lastFinishedPulling="2025-10-11 11:03:13.005137349 +0000 UTC m=+2224.979363790" observedRunningTime="2025-10-11 11:03:14.117686889 +0000 UTC m=+2226.091913330" watchObservedRunningTime="2025-10-11 11:03:14.122916259 +0000 UTC m=+2226.097142690" Oct 11 11:03:17.127735 master-1 kubenswrapper[4771]: I1011 11:03:17.127640 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-image-upload-678599687f-vbxnh" event={"ID":"30a9a1fe-e08c-4112-a7b8-6616d280405e","Type":"ContainerStarted","Data":"04f298e98d945615f593936f0afe3e98a8fda50794b9ca2f8c1e2dcce3be303c"} Oct 11 11:03:18.140216 master-1 kubenswrapper[4771]: I1011 11:03:18.140138 4771 generic.go:334] "Generic (PLEG): container finished" podID="30a9a1fe-e08c-4112-a7b8-6616d280405e" containerID="04f298e98d945615f593936f0afe3e98a8fda50794b9ca2f8c1e2dcce3be303c" exitCode=0 Oct 11 11:03:18.140216 master-1 kubenswrapper[4771]: I1011 11:03:18.140210 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-image-upload-678599687f-vbxnh" event={"ID":"30a9a1fe-e08c-4112-a7b8-6616d280405e","Type":"ContainerDied","Data":"04f298e98d945615f593936f0afe3e98a8fda50794b9ca2f8c1e2dcce3be303c"} Oct 11 11:03:18.977814 master-1 kubenswrapper[4771]: I1011 11:03:18.977733 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/octavia-rsyslog-ljmzz" Oct 11 11:03:19.152918 master-1 kubenswrapper[4771]: I1011 11:03:19.152773 4771 generic.go:334] "Generic (PLEG): container finished" podID="2681ddac-5e31-449b-bf71-fb54e8ba389c" containerID="00d99e2ba51100c415ae5d1a3b19ce4ab68cd0b4655796bec9fd8f7ec75f10f8" exitCode=0 Oct 11 11:03:19.153439 master-1 kubenswrapper[4771]: I1011 11:03:19.152902 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-db-sync-dsphh" event={"ID":"2681ddac-5e31-449b-bf71-fb54e8ba389c","Type":"ContainerDied","Data":"00d99e2ba51100c415ae5d1a3b19ce4ab68cd0b4655796bec9fd8f7ec75f10f8"} Oct 11 11:03:19.155054 master-1 kubenswrapper[4771]: I1011 11:03:19.154989 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-image-upload-678599687f-vbxnh" event={"ID":"30a9a1fe-e08c-4112-a7b8-6616d280405e","Type":"ContainerStarted","Data":"bb7f7c6b7f921511c546d7f35dc55b164c7677f19e59519f7cff36e97aa18bfd"} Oct 11 11:03:19.459883 master-1 kubenswrapper[4771]: I1011 11:03:19.459568 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/octavia-image-upload-678599687f-vbxnh" podStartSLOduration=4.038228471 podStartE2EDuration="15.459537673s" podCreationTimestamp="2025-10-11 11:03:04 +0000 UTC" firstStartedPulling="2025-10-11 11:03:05.293983706 +0000 UTC m=+2217.268210147" lastFinishedPulling="2025-10-11 11:03:16.715292868 +0000 UTC m=+2228.689519349" observedRunningTime="2025-10-11 11:03:19.45104481 +0000 UTC m=+2231.425271301" watchObservedRunningTime="2025-10-11 11:03:19.459537673 +0000 UTC m=+2231.433764154" Oct 11 11:03:20.702833 master-1 kubenswrapper[4771]: I1011 11:03:20.702774 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:03:20.741057 master-1 kubenswrapper[4771]: I1011 11:03:20.739845 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:03:20.746227 master-1 kubenswrapper[4771]: I1011 11:03:20.746141 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:20.880615 master-1 kubenswrapper[4771]: I1011 11:03:20.880478 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data-merged\") pod \"2681ddac-5e31-449b-bf71-fb54e8ba389c\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " Oct 11 11:03:20.880615 master-1 kubenswrapper[4771]: I1011 11:03:20.880585 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-scripts\") pod \"2681ddac-5e31-449b-bf71-fb54e8ba389c\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " Oct 11 11:03:20.880947 master-1 kubenswrapper[4771]: I1011 11:03:20.880695 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-combined-ca-bundle\") pod \"2681ddac-5e31-449b-bf71-fb54e8ba389c\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " Oct 11 11:03:20.880947 master-1 kubenswrapper[4771]: I1011 11:03:20.880785 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data\") pod \"2681ddac-5e31-449b-bf71-fb54e8ba389c\" (UID: \"2681ddac-5e31-449b-bf71-fb54e8ba389c\") " Oct 11 11:03:20.886405 master-1 kubenswrapper[4771]: I1011 11:03:20.886294 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-scripts" (OuterVolumeSpecName: "scripts") pod "2681ddac-5e31-449b-bf71-fb54e8ba389c" (UID: "2681ddac-5e31-449b-bf71-fb54e8ba389c"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:20.902183 master-1 kubenswrapper[4771]: I1011 11:03:20.902100 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data" (OuterVolumeSpecName: "config-data") pod "2681ddac-5e31-449b-bf71-fb54e8ba389c" (UID: "2681ddac-5e31-449b-bf71-fb54e8ba389c"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:20.902520 master-1 kubenswrapper[4771]: I1011 11:03:20.902446 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data-merged" (OuterVolumeSpecName: "config-data-merged") pod "2681ddac-5e31-449b-bf71-fb54e8ba389c" (UID: "2681ddac-5e31-449b-bf71-fb54e8ba389c"). InnerVolumeSpecName "config-data-merged". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:03:20.929628 master-1 kubenswrapper[4771]: I1011 11:03:20.929505 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "2681ddac-5e31-449b-bf71-fb54e8ba389c" (UID: "2681ddac-5e31-449b-bf71-fb54e8ba389c"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:20.983507 master-1 kubenswrapper[4771]: I1011 11:03:20.983272 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:20.983507 master-1 kubenswrapper[4771]: I1011 11:03:20.983318 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:20.983507 master-1 kubenswrapper[4771]: I1011 11:03:20.983331 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:20.983507 master-1 kubenswrapper[4771]: I1011 11:03:20.983340 4771 reconciler_common.go:293] "Volume detached for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/2681ddac-5e31-449b-bf71-fb54e8ba389c-config-data-merged\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:21.183288 master-1 kubenswrapper[4771]: I1011 11:03:21.183153 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-db-sync-dsphh" Oct 11 11:03:21.183536 master-1 kubenswrapper[4771]: I1011 11:03:21.183286 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-db-sync-dsphh" event={"ID":"2681ddac-5e31-449b-bf71-fb54e8ba389c","Type":"ContainerDied","Data":"62529d885f50e38d46775355d42d0bf6618c28a31161c124af0a288723c5173b"} Oct 11 11:03:21.183536 master-1 kubenswrapper[4771]: I1011 11:03:21.183333 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62529d885f50e38d46775355d42d0bf6618c28a31161c124af0a288723c5173b" Oct 11 11:03:22.010579 master-1 kubenswrapper[4771]: I1011 11:03:22.010421 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-api-7f8fd8dbdd-z9j79"] Oct 11 11:03:22.011268 master-1 kubenswrapper[4771]: E1011 11:03:22.010835 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2681ddac-5e31-449b-bf71-fb54e8ba389c" containerName="octavia-db-sync" Oct 11 11:03:22.011268 master-1 kubenswrapper[4771]: I1011 11:03:22.010870 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2681ddac-5e31-449b-bf71-fb54e8ba389c" containerName="octavia-db-sync" Oct 11 11:03:22.011268 master-1 kubenswrapper[4771]: E1011 11:03:22.010893 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="2681ddac-5e31-449b-bf71-fb54e8ba389c" containerName="init" Oct 11 11:03:22.011268 master-1 kubenswrapper[4771]: I1011 11:03:22.010903 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="2681ddac-5e31-449b-bf71-fb54e8ba389c" containerName="init" Oct 11 11:03:22.011268 master-1 kubenswrapper[4771]: I1011 11:03:22.011103 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="2681ddac-5e31-449b-bf71-fb54e8ba389c" containerName="octavia-db-sync" Oct 11 11:03:22.012647 master-1 kubenswrapper[4771]: I1011 11:03:22.012603 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.015814 master-1 kubenswrapper[4771]: I1011 11:03:22.015749 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-octavia-internal-svc" Oct 11 11:03:22.015983 master-1 kubenswrapper[4771]: I1011 11:03:22.015900 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-octavia-public-svc" Oct 11 11:03:22.042388 master-1 kubenswrapper[4771]: I1011 11:03:22.042264 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-api-7f8fd8dbdd-z9j79"] Oct 11 11:03:22.108376 master-1 kubenswrapper[4771]: I1011 11:03:22.108274 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-public-tls-certs\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.108631 master-1 kubenswrapper[4771]: I1011 11:03:22.108358 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-combined-ca-bundle\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.108631 master-1 kubenswrapper[4771]: I1011 11:03:22.108436 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"octavia-run\" (UniqueName: \"kubernetes.io/empty-dir/d7a14abd-dac6-484f-8be5-a64699a6318a-octavia-run\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.108631 master-1 kubenswrapper[4771]: I1011 11:03:22.108517 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-ovndb-tls-certs\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.108631 master-1 kubenswrapper[4771]: I1011 11:03:22.108571 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-scripts\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.108888 master-1 kubenswrapper[4771]: I1011 11:03:22.108700 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-internal-tls-certs\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.108888 master-1 kubenswrapper[4771]: I1011 11:03:22.108769 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-config-data\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.108888 master-1 kubenswrapper[4771]: I1011 11:03:22.108849 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/d7a14abd-dac6-484f-8be5-a64699a6318a-config-data-merged\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.218879 master-1 kubenswrapper[4771]: I1011 11:03:22.211261 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-ovndb-tls-certs\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.218879 master-1 kubenswrapper[4771]: I1011 11:03:22.211339 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-scripts\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.218879 master-1 kubenswrapper[4771]: I1011 11:03:22.211416 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-internal-tls-certs\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.218879 master-1 kubenswrapper[4771]: I1011 11:03:22.211476 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-config-data\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.218879 master-1 kubenswrapper[4771]: I1011 11:03:22.211613 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/d7a14abd-dac6-484f-8be5-a64699a6318a-config-data-merged\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.218879 master-1 kubenswrapper[4771]: I1011 11:03:22.212038 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-public-tls-certs\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.218879 master-1 kubenswrapper[4771]: I1011 11:03:22.212072 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-combined-ca-bundle\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.218879 master-1 kubenswrapper[4771]: I1011 11:03:22.212102 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"octavia-run\" (UniqueName: \"kubernetes.io/empty-dir/d7a14abd-dac6-484f-8be5-a64699a6318a-octavia-run\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.218879 master-1 kubenswrapper[4771]: I1011 11:03:22.213131 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"octavia-run\" (UniqueName: \"kubernetes.io/empty-dir/d7a14abd-dac6-484f-8be5-a64699a6318a-octavia-run\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.220062 master-1 kubenswrapper[4771]: I1011 11:03:22.219523 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-ovndb-tls-certs\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.220930 master-1 kubenswrapper[4771]: I1011 11:03:22.220883 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/d7a14abd-dac6-484f-8be5-a64699a6318a-config-data-merged\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.223584 master-1 kubenswrapper[4771]: I1011 11:03:22.223526 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-scripts\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.226798 master-1 kubenswrapper[4771]: I1011 11:03:22.226752 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"internal-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-internal-tls-certs\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.227696 master-1 kubenswrapper[4771]: I1011 11:03:22.227639 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"public-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-public-tls-certs\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.229177 master-1 kubenswrapper[4771]: I1011 11:03:22.229109 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-combined-ca-bundle\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.232381 master-1 kubenswrapper[4771]: I1011 11:03:22.232309 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d7a14abd-dac6-484f-8be5-a64699a6318a-config-data\") pod \"octavia-api-7f8fd8dbdd-z9j79\" (UID: \"d7a14abd-dac6-484f-8be5-a64699a6318a\") " pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.331815 master-1 kubenswrapper[4771]: I1011 11:03:22.331762 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:22.960808 master-1 kubenswrapper[4771]: I1011 11:03:22.960748 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-api-7f8fd8dbdd-z9j79"] Oct 11 11:03:23.239060 master-1 kubenswrapper[4771]: I1011 11:03:23.220512 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" event={"ID":"d7a14abd-dac6-484f-8be5-a64699a6318a","Type":"ContainerStarted","Data":"22fa22b2e1e0fff56977fc20f36b3003169ab500f911c05f2401cbd9c83f35f5"} Oct 11 11:03:23.239060 master-1 kubenswrapper[4771]: I1011 11:03:23.220592 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" event={"ID":"d7a14abd-dac6-484f-8be5-a64699a6318a","Type":"ContainerStarted","Data":"e641101a9ca2b11120834b9cee131298e55ecf3dc36dfa33ecae8239b2afd40d"} Oct 11 11:03:24.239576 master-1 kubenswrapper[4771]: I1011 11:03:24.237417 4771 generic.go:334] "Generic (PLEG): container finished" podID="d7a14abd-dac6-484f-8be5-a64699a6318a" containerID="22fa22b2e1e0fff56977fc20f36b3003169ab500f911c05f2401cbd9c83f35f5" exitCode=0 Oct 11 11:03:24.239576 master-1 kubenswrapper[4771]: I1011 11:03:24.237489 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" event={"ID":"d7a14abd-dac6-484f-8be5-a64699a6318a","Type":"ContainerDied","Data":"22fa22b2e1e0fff56977fc20f36b3003169ab500f911c05f2401cbd9c83f35f5"} Oct 11 11:03:24.287124 master-1 kubenswrapper[4771]: I1011 11:03:24.287031 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:24.287604 master-1 kubenswrapper[4771]: I1011 11:03:24.287545 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="ceilometer-central-agent" containerID="cri-o://2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea" gracePeriod=30 Oct 11 11:03:24.287923 master-1 kubenswrapper[4771]: I1011 11:03:24.287863 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="proxy-httpd" containerID="cri-o://d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f" gracePeriod=30 Oct 11 11:03:24.288143 master-1 kubenswrapper[4771]: I1011 11:03:24.288094 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="ceilometer-notification-agent" containerID="cri-o://f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692" gracePeriod=30 Oct 11 11:03:24.288670 master-1 kubenswrapper[4771]: I1011 11:03:24.288219 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/ceilometer-0" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="sg-core" containerID="cri-o://ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3" gracePeriod=30 Oct 11 11:03:24.399840 master-1 kubenswrapper[4771]: I1011 11:03:24.399744 4771 prober.go:107] "Probe failed" probeType="Readiness" pod="openstack/ceilometer-0" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="proxy-httpd" probeResult="failure" output="Get \"https://10.129.0.185:3000/\": read tcp 10.129.0.2:56100->10.129.0.185:3000: read: connection reset by peer" Oct 11 11:03:25.251194 master-1 kubenswrapper[4771]: I1011 11:03:25.251106 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" event={"ID":"d7a14abd-dac6-484f-8be5-a64699a6318a","Type":"ContainerStarted","Data":"79831f65d452f5ad8ac5b696a3cd986522fd67e265fdd4f945f48ec4a2ef905f"} Oct 11 11:03:25.251194 master-1 kubenswrapper[4771]: I1011 11:03:25.251177 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" event={"ID":"d7a14abd-dac6-484f-8be5-a64699a6318a","Type":"ContainerStarted","Data":"bf0aa14e0e1efa873d57b5787f324c374fc683b1feac774fc95901a1f4611c7f"} Oct 11 11:03:25.252178 master-1 kubenswrapper[4771]: I1011 11:03:25.251654 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:25.252178 master-1 kubenswrapper[4771]: I1011 11:03:25.251683 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:25.255381 master-1 kubenswrapper[4771]: I1011 11:03:25.255307 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerDied","Data":"d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f"} Oct 11 11:03:25.255792 master-1 kubenswrapper[4771]: I1011 11:03:25.255721 4771 generic.go:334] "Generic (PLEG): container finished" podID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerID="d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f" exitCode=0 Oct 11 11:03:25.255792 master-1 kubenswrapper[4771]: I1011 11:03:25.255785 4771 generic.go:334] "Generic (PLEG): container finished" podID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerID="ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3" exitCode=2 Oct 11 11:03:25.256005 master-1 kubenswrapper[4771]: I1011 11:03:25.255809 4771 generic.go:334] "Generic (PLEG): container finished" podID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerID="2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea" exitCode=0 Oct 11 11:03:25.256005 master-1 kubenswrapper[4771]: I1011 11:03:25.255845 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerDied","Data":"ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3"} Oct 11 11:03:25.256005 master-1 kubenswrapper[4771]: I1011 11:03:25.255881 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerDied","Data":"2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea"} Oct 11 11:03:25.299486 master-1 kubenswrapper[4771]: I1011 11:03:25.298865 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" podStartSLOduration=4.29878046 podStartE2EDuration="4.29878046s" podCreationTimestamp="2025-10-11 11:03:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 11:03:25.293677803 +0000 UTC m=+2237.267904274" watchObservedRunningTime="2025-10-11 11:03:25.29878046 +0000 UTC m=+2237.273006901" Oct 11 11:03:26.176242 master-1 kubenswrapper[4771]: I1011 11:03:26.176132 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 11:03:26.280040 master-1 kubenswrapper[4771]: I1011 11:03:26.279958 4771 generic.go:334] "Generic (PLEG): container finished" podID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerID="f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692" exitCode=0 Oct 11 11:03:26.280040 master-1 kubenswrapper[4771]: I1011 11:03:26.280035 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 11:03:26.280992 master-1 kubenswrapper[4771]: I1011 11:03:26.280046 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerDied","Data":"f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692"} Oct 11 11:03:26.280992 master-1 kubenswrapper[4771]: I1011 11:03:26.280311 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d199f8f0-ae54-4c5c-b8af-f3058547edf1","Type":"ContainerDied","Data":"1f5edb60e620fedfd68b508b4b511f9365534396f20ac698191ef7d2d59c6bfb"} Oct 11 11:03:26.280992 master-1 kubenswrapper[4771]: I1011 11:03:26.280338 4771 scope.go:117] "RemoveContainer" containerID="d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f" Oct 11 11:03:26.304770 master-1 kubenswrapper[4771]: I1011 11:03:26.304593 4771 scope.go:117] "RemoveContainer" containerID="ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3" Oct 11 11:03:26.321543 master-1 kubenswrapper[4771]: I1011 11:03:26.321487 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-config-data\") pod \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " Oct 11 11:03:26.321543 master-1 kubenswrapper[4771]: I1011 11:03:26.321543 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-run-httpd\") pod \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " Oct 11 11:03:26.322002 master-1 kubenswrapper[4771]: I1011 11:03:26.321672 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc67p\" (UniqueName: \"kubernetes.io/projected/d199f8f0-ae54-4c5c-b8af-f3058547edf1-kube-api-access-jc67p\") pod \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " Oct 11 11:03:26.322002 master-1 kubenswrapper[4771]: I1011 11:03:26.321760 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-ceilometer-tls-certs\") pod \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " Oct 11 11:03:26.322002 master-1 kubenswrapper[4771]: I1011 11:03:26.321870 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-sg-core-conf-yaml\") pod \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " Oct 11 11:03:26.322002 master-1 kubenswrapper[4771]: I1011 11:03:26.321911 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-combined-ca-bundle\") pod \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " Oct 11 11:03:26.322002 master-1 kubenswrapper[4771]: I1011 11:03:26.321957 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-scripts\") pod \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " Oct 11 11:03:26.322002 master-1 kubenswrapper[4771]: I1011 11:03:26.321975 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-log-httpd\") pod \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\" (UID: \"d199f8f0-ae54-4c5c-b8af-f3058547edf1\") " Oct 11 11:03:26.322288 master-1 kubenswrapper[4771]: I1011 11:03:26.322189 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-run-httpd" (OuterVolumeSpecName: "run-httpd") pod "d199f8f0-ae54-4c5c-b8af-f3058547edf1" (UID: "d199f8f0-ae54-4c5c-b8af-f3058547edf1"). InnerVolumeSpecName "run-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:03:26.324202 master-1 kubenswrapper[4771]: I1011 11:03:26.323807 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-log-httpd" (OuterVolumeSpecName: "log-httpd") pod "d199f8f0-ae54-4c5c-b8af-f3058547edf1" (UID: "d199f8f0-ae54-4c5c-b8af-f3058547edf1"). InnerVolumeSpecName "log-httpd". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:03:26.324202 master-1 kubenswrapper[4771]: I1011 11:03:26.324123 4771 reconciler_common.go:293] "Volume detached for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-log-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:26.324202 master-1 kubenswrapper[4771]: I1011 11:03:26.324167 4771 reconciler_common.go:293] "Volume detached for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d199f8f0-ae54-4c5c-b8af-f3058547edf1-run-httpd\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:26.327763 master-1 kubenswrapper[4771]: I1011 11:03:26.327730 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-scripts" (OuterVolumeSpecName: "scripts") pod "d199f8f0-ae54-4c5c-b8af-f3058547edf1" (UID: "d199f8f0-ae54-4c5c-b8af-f3058547edf1"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:26.329484 master-1 kubenswrapper[4771]: I1011 11:03:26.329437 4771 scope.go:117] "RemoveContainer" containerID="f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692" Oct 11 11:03:26.337333 master-1 kubenswrapper[4771]: I1011 11:03:26.337244 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d199f8f0-ae54-4c5c-b8af-f3058547edf1-kube-api-access-jc67p" (OuterVolumeSpecName: "kube-api-access-jc67p") pod "d199f8f0-ae54-4c5c-b8af-f3058547edf1" (UID: "d199f8f0-ae54-4c5c-b8af-f3058547edf1"). InnerVolumeSpecName "kube-api-access-jc67p". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:03:26.350705 master-1 kubenswrapper[4771]: I1011 11:03:26.350619 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-sg-core-conf-yaml" (OuterVolumeSpecName: "sg-core-conf-yaml") pod "d199f8f0-ae54-4c5c-b8af-f3058547edf1" (UID: "d199f8f0-ae54-4c5c-b8af-f3058547edf1"). InnerVolumeSpecName "sg-core-conf-yaml". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:26.396676 master-1 kubenswrapper[4771]: I1011 11:03:26.396456 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-ceilometer-tls-certs" (OuterVolumeSpecName: "ceilometer-tls-certs") pod "d199f8f0-ae54-4c5c-b8af-f3058547edf1" (UID: "d199f8f0-ae54-4c5c-b8af-f3058547edf1"). InnerVolumeSpecName "ceilometer-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:26.413245 master-1 kubenswrapper[4771]: I1011 11:03:26.413161 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-config-data" (OuterVolumeSpecName: "config-data") pod "d199f8f0-ae54-4c5c-b8af-f3058547edf1" (UID: "d199f8f0-ae54-4c5c-b8af-f3058547edf1"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:26.413734 master-1 kubenswrapper[4771]: I1011 11:03:26.413665 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "d199f8f0-ae54-4c5c-b8af-f3058547edf1" (UID: "d199f8f0-ae54-4c5c-b8af-f3058547edf1"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:26.426032 master-1 kubenswrapper[4771]: I1011 11:03:26.425957 4771 reconciler_common.go:293] "Volume detached for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-ceilometer-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:26.426032 master-1 kubenswrapper[4771]: I1011 11:03:26.426013 4771 reconciler_common.go:293] "Volume detached for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-sg-core-conf-yaml\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:26.426032 master-1 kubenswrapper[4771]: I1011 11:03:26.426029 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:26.426032 master-1 kubenswrapper[4771]: I1011 11:03:26.426046 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:26.426032 master-1 kubenswrapper[4771]: I1011 11:03:26.426062 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d199f8f0-ae54-4c5c-b8af-f3058547edf1-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:26.426671 master-1 kubenswrapper[4771]: I1011 11:03:26.426076 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-jc67p\" (UniqueName: \"kubernetes.io/projected/d199f8f0-ae54-4c5c-b8af-f3058547edf1-kube-api-access-jc67p\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:26.472098 master-1 kubenswrapper[4771]: I1011 11:03:26.472053 4771 scope.go:117] "RemoveContainer" containerID="2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea" Oct 11 11:03:26.495745 master-1 kubenswrapper[4771]: I1011 11:03:26.495693 4771 scope.go:117] "RemoveContainer" containerID="d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f" Oct 11 11:03:26.496543 master-1 kubenswrapper[4771]: E1011 11:03:26.496505 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f\": container with ID starting with d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f not found: ID does not exist" containerID="d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f" Oct 11 11:03:26.496601 master-1 kubenswrapper[4771]: I1011 11:03:26.496544 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f"} err="failed to get container status \"d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f\": rpc error: code = NotFound desc = could not find container \"d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f\": container with ID starting with d509506178d765ed7f4a16904a37ddc1b2c54d25ae15ba1153670674913b8d5f not found: ID does not exist" Oct 11 11:03:26.496601 master-1 kubenswrapper[4771]: I1011 11:03:26.496567 4771 scope.go:117] "RemoveContainer" containerID="ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3" Oct 11 11:03:26.497079 master-1 kubenswrapper[4771]: E1011 11:03:26.497047 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3\": container with ID starting with ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3 not found: ID does not exist" containerID="ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3" Oct 11 11:03:26.497133 master-1 kubenswrapper[4771]: I1011 11:03:26.497080 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3"} err="failed to get container status \"ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3\": rpc error: code = NotFound desc = could not find container \"ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3\": container with ID starting with ca41f80532d4f090e18655608cd8c3f3de6ab84ef57c4a67c84ac53396fb2ce3 not found: ID does not exist" Oct 11 11:03:26.497133 master-1 kubenswrapper[4771]: I1011 11:03:26.497095 4771 scope.go:117] "RemoveContainer" containerID="f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692" Oct 11 11:03:26.497591 master-1 kubenswrapper[4771]: E1011 11:03:26.497553 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692\": container with ID starting with f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692 not found: ID does not exist" containerID="f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692" Oct 11 11:03:26.497591 master-1 kubenswrapper[4771]: I1011 11:03:26.497585 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692"} err="failed to get container status \"f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692\": rpc error: code = NotFound desc = could not find container \"f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692\": container with ID starting with f369b17be5d671fc3e32876475f80c5b0218179c8afd4aa1df1108a929725692 not found: ID does not exist" Oct 11 11:03:26.497684 master-1 kubenswrapper[4771]: I1011 11:03:26.497600 4771 scope.go:117] "RemoveContainer" containerID="2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea" Oct 11 11:03:26.499913 master-1 kubenswrapper[4771]: E1011 11:03:26.499860 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea\": container with ID starting with 2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea not found: ID does not exist" containerID="2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea" Oct 11 11:03:26.499989 master-1 kubenswrapper[4771]: I1011 11:03:26.499921 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea"} err="failed to get container status \"2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea\": rpc error: code = NotFound desc = could not find container \"2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea\": container with ID starting with 2e42285e6351eb9ee62ba992f1e4f7be63e97a858d54b24d10c71afdf7aff3ea not found: ID does not exist" Oct 11 11:03:26.612312 master-1 kubenswrapper[4771]: I1011 11:03:26.612232 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:26.621999 master-1 kubenswrapper[4771]: I1011 11:03:26.621958 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:26.651595 master-1 kubenswrapper[4771]: I1011 11:03:26.651235 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:26.652235 master-1 kubenswrapper[4771]: E1011 11:03:26.652213 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="sg-core" Oct 11 11:03:26.652352 master-1 kubenswrapper[4771]: I1011 11:03:26.652339 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="sg-core" Oct 11 11:03:26.652461 master-1 kubenswrapper[4771]: E1011 11:03:26.652449 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="proxy-httpd" Oct 11 11:03:26.652517 master-1 kubenswrapper[4771]: I1011 11:03:26.652508 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="proxy-httpd" Oct 11 11:03:26.652593 master-1 kubenswrapper[4771]: E1011 11:03:26.652581 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="ceilometer-central-agent" Oct 11 11:03:26.652682 master-1 kubenswrapper[4771]: I1011 11:03:26.652668 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="ceilometer-central-agent" Oct 11 11:03:26.652796 master-1 kubenswrapper[4771]: E1011 11:03:26.652780 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="ceilometer-notification-agent" Oct 11 11:03:26.652932 master-1 kubenswrapper[4771]: I1011 11:03:26.652913 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="ceilometer-notification-agent" Oct 11 11:03:26.653369 master-1 kubenswrapper[4771]: I1011 11:03:26.653150 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="sg-core" Oct 11 11:03:26.653523 master-1 kubenswrapper[4771]: I1011 11:03:26.653509 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="proxy-httpd" Oct 11 11:03:26.653854 master-1 kubenswrapper[4771]: I1011 11:03:26.653833 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="ceilometer-central-agent" Oct 11 11:03:26.653932 master-1 kubenswrapper[4771]: I1011 11:03:26.653922 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" containerName="ceilometer-notification-agent" Oct 11 11:03:26.655616 master-1 kubenswrapper[4771]: I1011 11:03:26.655598 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 11:03:26.661445 master-1 kubenswrapper[4771]: I1011 11:03:26.661342 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-config-data" Oct 11 11:03:26.661878 master-1 kubenswrapper[4771]: I1011 11:03:26.661849 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"cert-ceilometer-internal-svc" Oct 11 11:03:26.662044 master-1 kubenswrapper[4771]: I1011 11:03:26.662020 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-scripts" Oct 11 11:03:26.671432 master-1 kubenswrapper[4771]: I1011 11:03:26.671373 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:26.755016 master-1 kubenswrapper[4771]: I1011 11:03:26.754969 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.755488 master-1 kubenswrapper[4771]: I1011 11:03:26.755463 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gtk\" (UniqueName: \"kubernetes.io/projected/d247640d-5d67-4ba9-a371-0aa12cc122c6-kube-api-access-p5gtk\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.755602 master-1 kubenswrapper[4771]: I1011 11:03:26.755586 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d247640d-5d67-4ba9-a371-0aa12cc122c6-run-httpd\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.755702 master-1 kubenswrapper[4771]: I1011 11:03:26.755690 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-scripts\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.755789 master-1 kubenswrapper[4771]: I1011 11:03:26.755777 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.755897 master-1 kubenswrapper[4771]: I1011 11:03:26.755877 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d247640d-5d67-4ba9-a371-0aa12cc122c6-log-httpd\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.756082 master-1 kubenswrapper[4771]: I1011 11:03:26.756063 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.756416 master-1 kubenswrapper[4771]: I1011 11:03:26.756260 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-config-data\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.858110 master-1 kubenswrapper[4771]: I1011 11:03:26.858073 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-config-data\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.858892 master-1 kubenswrapper[4771]: I1011 11:03:26.858872 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.859021 master-1 kubenswrapper[4771]: I1011 11:03:26.859005 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gtk\" (UniqueName: \"kubernetes.io/projected/d247640d-5d67-4ba9-a371-0aa12cc122c6-kube-api-access-p5gtk\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.859157 master-1 kubenswrapper[4771]: I1011 11:03:26.859143 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d247640d-5d67-4ba9-a371-0aa12cc122c6-run-httpd\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.859259 master-1 kubenswrapper[4771]: I1011 11:03:26.859246 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-scripts\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.859405 master-1 kubenswrapper[4771]: I1011 11:03:26.859391 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.859512 master-1 kubenswrapper[4771]: I1011 11:03:26.859497 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d247640d-5d67-4ba9-a371-0aa12cc122c6-log-httpd\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.859659 master-1 kubenswrapper[4771]: I1011 11:03:26.859642 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.860238 master-1 kubenswrapper[4771]: I1011 11:03:26.860183 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"run-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d247640d-5d67-4ba9-a371-0aa12cc122c6-run-httpd\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.860688 master-1 kubenswrapper[4771]: I1011 11:03:26.860634 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"log-httpd\" (UniqueName: \"kubernetes.io/empty-dir/d247640d-5d67-4ba9-a371-0aa12cc122c6-log-httpd\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.863149 master-1 kubenswrapper[4771]: I1011 11:03:26.863123 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ceilometer-tls-certs\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-ceilometer-tls-certs\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.863721 master-1 kubenswrapper[4771]: I1011 11:03:26.863504 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sg-core-conf-yaml\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-sg-core-conf-yaml\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.864568 master-1 kubenswrapper[4771]: I1011 11:03:26.864543 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-scripts\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.864689 master-1 kubenswrapper[4771]: I1011 11:03:26.864601 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-combined-ca-bundle\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.866186 master-1 kubenswrapper[4771]: I1011 11:03:26.866132 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/d247640d-5d67-4ba9-a371-0aa12cc122c6-config-data\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.888925 master-1 kubenswrapper[4771]: I1011 11:03:26.888891 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gtk\" (UniqueName: \"kubernetes.io/projected/d247640d-5d67-4ba9-a371-0aa12cc122c6-kube-api-access-p5gtk\") pod \"ceilometer-0\" (UID: \"d247640d-5d67-4ba9-a371-0aa12cc122c6\") " pod="openstack/ceilometer-0" Oct 11 11:03:26.979575 master-1 kubenswrapper[4771]: I1011 11:03:26.979470 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ceilometer-0" Oct 11 11:03:27.467856 master-1 kubenswrapper[4771]: I1011 11:03:27.467748 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ceilometer-0"] Oct 11 11:03:28.319672 master-1 kubenswrapper[4771]: I1011 11:03:28.319615 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d247640d-5d67-4ba9-a371-0aa12cc122c6","Type":"ContainerStarted","Data":"1a1b47943d68bc0445b87a7d7de76ea6131c5497e89e624301640c77c81a4800"} Oct 11 11:03:28.451718 master-1 kubenswrapper[4771]: I1011 11:03:28.451626 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d199f8f0-ae54-4c5c-b8af-f3058547edf1" path="/var/lib/kubelet/pods/d199f8f0-ae54-4c5c-b8af-f3058547edf1/volumes" Oct 11 11:03:29.328506 master-1 kubenswrapper[4771]: I1011 11:03:29.328435 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d247640d-5d67-4ba9-a371-0aa12cc122c6","Type":"ContainerStarted","Data":"423d9d672627475e5bbe8504c353fff88e7d057fbbe5abd832cfbec0141ccb20"} Oct 11 11:03:30.342440 master-1 kubenswrapper[4771]: I1011 11:03:30.341904 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d247640d-5d67-4ba9-a371-0aa12cc122c6","Type":"ContainerStarted","Data":"c280fe9bdf8d1af8f1dcc4147c2d8952707f98e0ff2dfb05da03cdcf2eb663a1"} Oct 11 11:03:32.378616 master-1 kubenswrapper[4771]: I1011 11:03:32.378524 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d247640d-5d67-4ba9-a371-0aa12cc122c6","Type":"ContainerStarted","Data":"76136259c130fdcf90abb03fcc84dd6667ae2aab8b9be639ce6e95fb5ec43ca0"} Oct 11 11:03:33.391156 master-1 kubenswrapper[4771]: I1011 11:03:33.391089 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ceilometer-0" event={"ID":"d247640d-5d67-4ba9-a371-0aa12cc122c6","Type":"ContainerStarted","Data":"b39399b5d8b4910fc52ba7ebd88e058e6f28aa63e2d7a7745dfc5b1297f52216"} Oct 11 11:03:33.391929 master-1 kubenswrapper[4771]: I1011 11:03:33.391328 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/ceilometer-0" Oct 11 11:03:33.427375 master-1 kubenswrapper[4771]: I1011 11:03:33.427258 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ceilometer-0" podStartSLOduration=2.052192018 podStartE2EDuration="7.427237196s" podCreationTimestamp="2025-10-11 11:03:26 +0000 UTC" firstStartedPulling="2025-10-11 11:03:27.483644926 +0000 UTC m=+2239.457871367" lastFinishedPulling="2025-10-11 11:03:32.858690104 +0000 UTC m=+2244.832916545" observedRunningTime="2025-10-11 11:03:33.423170069 +0000 UTC m=+2245.397396530" watchObservedRunningTime="2025-10-11 11:03:33.427237196 +0000 UTC m=+2245.401463647" Oct 11 11:03:41.255583 master-1 kubenswrapper[4771]: I1011 11:03:41.255499 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:41.325583 master-1 kubenswrapper[4771]: I1011 11:03:41.325527 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/octavia-api-7f8fd8dbdd-z9j79" Oct 11 11:03:41.605298 master-1 kubenswrapper[4771]: I1011 11:03:41.605231 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/octavia-api-84f885c68-ttgvk"] Oct 11 11:03:41.606276 master-1 kubenswrapper[4771]: I1011 11:03:41.605582 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/octavia-api-84f885c68-ttgvk" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="octavia-api-provider-agent" containerID="cri-o://e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598" gracePeriod=30 Oct 11 11:03:41.606276 master-1 kubenswrapper[4771]: I1011 11:03:41.605778 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/octavia-api-84f885c68-ttgvk" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="octavia-api" containerID="cri-o://4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530" gracePeriod=30 Oct 11 11:03:42.057913 master-1 kubenswrapper[4771]: I1011 11:03:42.057835 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/octavia-image-upload-678599687f-vbxnh"] Oct 11 11:03:42.058259 master-1 kubenswrapper[4771]: I1011 11:03:42.058202 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/octavia-image-upload-678599687f-vbxnh" podUID="30a9a1fe-e08c-4112-a7b8-6616d280405e" containerName="octavia-amphora-httpd" containerID="cri-o://bb7f7c6b7f921511c546d7f35dc55b164c7677f19e59519f7cff36e97aa18bfd" gracePeriod=30 Oct 11 11:03:42.507557 master-1 kubenswrapper[4771]: I1011 11:03:42.507468 4771 generic.go:334] "Generic (PLEG): container finished" podID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerID="e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598" exitCode=0 Oct 11 11:03:42.507557 master-1 kubenswrapper[4771]: I1011 11:03:42.507538 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-84f885c68-ttgvk" event={"ID":"a67756a2-aa42-4f6f-b27a-57e16f566883","Type":"ContainerDied","Data":"e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598"} Oct 11 11:03:42.514935 master-1 kubenswrapper[4771]: I1011 11:03:42.514871 4771 generic.go:334] "Generic (PLEG): container finished" podID="30a9a1fe-e08c-4112-a7b8-6616d280405e" containerID="bb7f7c6b7f921511c546d7f35dc55b164c7677f19e59519f7cff36e97aa18bfd" exitCode=0 Oct 11 11:03:42.515238 master-1 kubenswrapper[4771]: I1011 11:03:42.514944 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-image-upload-678599687f-vbxnh" event={"ID":"30a9a1fe-e08c-4112-a7b8-6616d280405e","Type":"ContainerDied","Data":"bb7f7c6b7f921511c546d7f35dc55b164c7677f19e59519f7cff36e97aa18bfd"} Oct 11 11:03:42.798951 master-1 kubenswrapper[4771]: I1011 11:03:42.798875 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:42.981074 master-1 kubenswrapper[4771]: I1011 11:03:42.980913 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"amphora-image\" (UniqueName: \"kubernetes.io/empty-dir/30a9a1fe-e08c-4112-a7b8-6616d280405e-amphora-image\") pod \"30a9a1fe-e08c-4112-a7b8-6616d280405e\" (UID: \"30a9a1fe-e08c-4112-a7b8-6616d280405e\") " Oct 11 11:03:42.981074 master-1 kubenswrapper[4771]: I1011 11:03:42.981053 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/30a9a1fe-e08c-4112-a7b8-6616d280405e-httpd-config\") pod \"30a9a1fe-e08c-4112-a7b8-6616d280405e\" (UID: \"30a9a1fe-e08c-4112-a7b8-6616d280405e\") " Oct 11 11:03:43.026216 master-1 kubenswrapper[4771]: I1011 11:03:43.026135 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a9a1fe-e08c-4112-a7b8-6616d280405e-httpd-config" (OuterVolumeSpecName: "httpd-config") pod "30a9a1fe-e08c-4112-a7b8-6616d280405e" (UID: "30a9a1fe-e08c-4112-a7b8-6616d280405e"). InnerVolumeSpecName "httpd-config". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:43.059028 master-1 kubenswrapper[4771]: I1011 11:03:43.058948 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/glance-db-create-dnsz5"] Oct 11 11:03:43.069797 master-1 kubenswrapper[4771]: I1011 11:03:43.069727 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/glance-db-create-dnsz5"] Oct 11 11:03:43.083678 master-1 kubenswrapper[4771]: I1011 11:03:43.083594 4771 reconciler_common.go:293] "Volume detached for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/30a9a1fe-e08c-4112-a7b8-6616d280405e-httpd-config\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:43.087754 master-1 kubenswrapper[4771]: I1011 11:03:43.087685 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a9a1fe-e08c-4112-a7b8-6616d280405e-amphora-image" (OuterVolumeSpecName: "amphora-image") pod "30a9a1fe-e08c-4112-a7b8-6616d280405e" (UID: "30a9a1fe-e08c-4112-a7b8-6616d280405e"). InnerVolumeSpecName "amphora-image". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:03:43.185587 master-1 kubenswrapper[4771]: I1011 11:03:43.185520 4771 reconciler_common.go:293] "Volume detached for volume \"amphora-image\" (UniqueName: \"kubernetes.io/empty-dir/30a9a1fe-e08c-4112-a7b8-6616d280405e-amphora-image\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:43.534117 master-1 kubenswrapper[4771]: I1011 11:03:43.533994 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-image-upload-678599687f-vbxnh" event={"ID":"30a9a1fe-e08c-4112-a7b8-6616d280405e","Type":"ContainerDied","Data":"822d91bade265914abdf5fc87f453e2f88a3133976d26e99e2272117f587739c"} Oct 11 11:03:43.534117 master-1 kubenswrapper[4771]: I1011 11:03:43.534088 4771 scope.go:117] "RemoveContainer" containerID="bb7f7c6b7f921511c546d7f35dc55b164c7677f19e59519f7cff36e97aa18bfd" Oct 11 11:03:43.535385 master-1 kubenswrapper[4771]: I1011 11:03:43.534290 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-image-upload-678599687f-vbxnh" Oct 11 11:03:43.581041 master-1 kubenswrapper[4771]: I1011 11:03:43.580994 4771 scope.go:117] "RemoveContainer" containerID="04f298e98d945615f593936f0afe3e98a8fda50794b9ca2f8c1e2dcce3be303c" Oct 11 11:03:43.604577 master-1 kubenswrapper[4771]: I1011 11:03:43.604489 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/octavia-image-upload-678599687f-vbxnh"] Oct 11 11:03:43.627024 master-1 kubenswrapper[4771]: I1011 11:03:43.626930 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/octavia-image-upload-678599687f-vbxnh"] Oct 11 11:03:44.454919 master-1 kubenswrapper[4771]: I1011 11:03:44.454844 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a9a1fe-e08c-4112-a7b8-6616d280405e" path="/var/lib/kubelet/pods/30a9a1fe-e08c-4112-a7b8-6616d280405e/volumes" Oct 11 11:03:44.455678 master-1 kubenswrapper[4771]: I1011 11:03:44.455652 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ac9130-d850-4420-a75e-53ec744b16eb" path="/var/lib/kubelet/pods/57ac9130-d850-4420-a75e-53ec744b16eb/volumes" Oct 11 11:03:45.494061 master-1 kubenswrapper[4771]: I1011 11:03:45.494009 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:03:45.556707 master-1 kubenswrapper[4771]: I1011 11:03:45.556632 4771 generic.go:334] "Generic (PLEG): container finished" podID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerID="4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530" exitCode=0 Oct 11 11:03:45.556707 master-1 kubenswrapper[4771]: I1011 11:03:45.556686 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-84f885c68-ttgvk" event={"ID":"a67756a2-aa42-4f6f-b27a-57e16f566883","Type":"ContainerDied","Data":"4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530"} Oct 11 11:03:45.556707 master-1 kubenswrapper[4771]: I1011 11:03:45.556725 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-api-84f885c68-ttgvk" event={"ID":"a67756a2-aa42-4f6f-b27a-57e16f566883","Type":"ContainerDied","Data":"a5a886249a2d7b53c7a4945ae527e497dd08083a42ca3105272e41c3a3ba5d39"} Oct 11 11:03:45.557147 master-1 kubenswrapper[4771]: I1011 11:03:45.556727 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-api-84f885c68-ttgvk" Oct 11 11:03:45.557147 master-1 kubenswrapper[4771]: I1011 11:03:45.556746 4771 scope.go:117] "RemoveContainer" containerID="e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598" Oct 11 11:03:45.580075 master-1 kubenswrapper[4771]: I1011 11:03:45.580003 4771 scope.go:117] "RemoveContainer" containerID="4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530" Oct 11 11:03:45.603607 master-1 kubenswrapper[4771]: I1011 11:03:45.603552 4771 scope.go:117] "RemoveContainer" containerID="36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c" Oct 11 11:03:45.628117 master-1 kubenswrapper[4771]: I1011 11:03:45.628083 4771 scope.go:117] "RemoveContainer" containerID="e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598" Oct 11 11:03:45.628570 master-1 kubenswrapper[4771]: E1011 11:03:45.628542 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598\": container with ID starting with e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598 not found: ID does not exist" containerID="e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598" Oct 11 11:03:45.628748 master-1 kubenswrapper[4771]: I1011 11:03:45.628582 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598"} err="failed to get container status \"e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598\": rpc error: code = NotFound desc = could not find container \"e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598\": container with ID starting with e41aadf1122cacc740648e97ac9f4e96f2409d4d65d6150033e5ce9dfc2c1598 not found: ID does not exist" Oct 11 11:03:45.628748 master-1 kubenswrapper[4771]: I1011 11:03:45.628610 4771 scope.go:117] "RemoveContainer" containerID="4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530" Oct 11 11:03:45.629457 master-1 kubenswrapper[4771]: E1011 11:03:45.629418 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530\": container with ID starting with 4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530 not found: ID does not exist" containerID="4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530" Oct 11 11:03:45.629607 master-1 kubenswrapper[4771]: I1011 11:03:45.629581 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530"} err="failed to get container status \"4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530\": rpc error: code = NotFound desc = could not find container \"4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530\": container with ID starting with 4b1b4afdacfba7ecd1fb1011c6d887849fae4d85214d8f2c59893edc87098530 not found: ID does not exist" Oct 11 11:03:45.629700 master-1 kubenswrapper[4771]: I1011 11:03:45.629685 4771 scope.go:117] "RemoveContainer" containerID="36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c" Oct 11 11:03:45.630104 master-1 kubenswrapper[4771]: E1011 11:03:45.630077 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c\": container with ID starting with 36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c not found: ID does not exist" containerID="36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c" Oct 11 11:03:45.630177 master-1 kubenswrapper[4771]: I1011 11:03:45.630105 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c"} err="failed to get container status \"36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c\": rpc error: code = NotFound desc = could not find container \"36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c\": container with ID starting with 36a2a93821ed51d9e67c4b879d36463d95dbf4ca7d7ecbdb8119182e791d541c not found: ID does not exist" Oct 11 11:03:45.657135 master-1 kubenswrapper[4771]: I1011 11:03:45.657004 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data\") pod \"a67756a2-aa42-4f6f-b27a-57e16f566883\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " Oct 11 11:03:45.657135 master-1 kubenswrapper[4771]: I1011 11:03:45.657101 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"octavia-run\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-octavia-run\") pod \"a67756a2-aa42-4f6f-b27a-57e16f566883\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " Oct 11 11:03:45.657135 master-1 kubenswrapper[4771]: I1011 11:03:45.657119 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-ovndb-tls-certs\") pod \"a67756a2-aa42-4f6f-b27a-57e16f566883\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " Oct 11 11:03:45.657442 master-1 kubenswrapper[4771]: I1011 11:03:45.657266 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-scripts\") pod \"a67756a2-aa42-4f6f-b27a-57e16f566883\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " Oct 11 11:03:45.657442 master-1 kubenswrapper[4771]: I1011 11:03:45.657307 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-combined-ca-bundle\") pod \"a67756a2-aa42-4f6f-b27a-57e16f566883\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " Oct 11 11:03:45.657442 master-1 kubenswrapper[4771]: I1011 11:03:45.657402 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data-merged\") pod \"a67756a2-aa42-4f6f-b27a-57e16f566883\" (UID: \"a67756a2-aa42-4f6f-b27a-57e16f566883\") " Oct 11 11:03:45.657635 master-1 kubenswrapper[4771]: I1011 11:03:45.657499 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-octavia-run" (OuterVolumeSpecName: "octavia-run") pod "a67756a2-aa42-4f6f-b27a-57e16f566883" (UID: "a67756a2-aa42-4f6f-b27a-57e16f566883"). InnerVolumeSpecName "octavia-run". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:03:45.657917 master-1 kubenswrapper[4771]: I1011 11:03:45.657895 4771 reconciler_common.go:293] "Volume detached for volume \"octavia-run\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-octavia-run\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:45.663658 master-1 kubenswrapper[4771]: I1011 11:03:45.663618 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data" (OuterVolumeSpecName: "config-data") pod "a67756a2-aa42-4f6f-b27a-57e16f566883" (UID: "a67756a2-aa42-4f6f-b27a-57e16f566883"). InnerVolumeSpecName "config-data". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:45.663808 master-1 kubenswrapper[4771]: I1011 11:03:45.663732 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-scripts" (OuterVolumeSpecName: "scripts") pod "a67756a2-aa42-4f6f-b27a-57e16f566883" (UID: "a67756a2-aa42-4f6f-b27a-57e16f566883"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:45.698729 master-1 kubenswrapper[4771]: I1011 11:03:45.698557 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data-merged" (OuterVolumeSpecName: "config-data-merged") pod "a67756a2-aa42-4f6f-b27a-57e16f566883" (UID: "a67756a2-aa42-4f6f-b27a-57e16f566883"). InnerVolumeSpecName "config-data-merged". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:03:45.710561 master-1 kubenswrapper[4771]: I1011 11:03:45.710491 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-combined-ca-bundle" (OuterVolumeSpecName: "combined-ca-bundle") pod "a67756a2-aa42-4f6f-b27a-57e16f566883" (UID: "a67756a2-aa42-4f6f-b27a-57e16f566883"). InnerVolumeSpecName "combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:45.762561 master-1 kubenswrapper[4771]: I1011 11:03:45.762333 4771 reconciler_common.go:293] "Volume detached for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:45.762561 master-1 kubenswrapper[4771]: I1011 11:03:45.762540 4771 reconciler_common.go:293] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-scripts\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:45.762561 master-1 kubenswrapper[4771]: I1011 11:03:45.762567 4771 reconciler_common.go:293] "Volume detached for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:45.762897 master-1 kubenswrapper[4771]: I1011 11:03:45.762592 4771 reconciler_common.go:293] "Volume detached for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/a67756a2-aa42-4f6f-b27a-57e16f566883-config-data-merged\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:45.779456 master-1 kubenswrapper[4771]: I1011 11:03:45.779395 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-ovndb-tls-certs" (OuterVolumeSpecName: "ovndb-tls-certs") pod "a67756a2-aa42-4f6f-b27a-57e16f566883" (UID: "a67756a2-aa42-4f6f-b27a-57e16f566883"). InnerVolumeSpecName "ovndb-tls-certs". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:03:45.865446 master-1 kubenswrapper[4771]: I1011 11:03:45.865328 4771 reconciler_common.go:293] "Volume detached for volume \"ovndb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a67756a2-aa42-4f6f-b27a-57e16f566883-ovndb-tls-certs\") on node \"master-1\" DevicePath \"\"" Oct 11 11:03:45.964070 master-1 kubenswrapper[4771]: I1011 11:03:45.963962 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/octavia-api-84f885c68-ttgvk"] Oct 11 11:03:45.971942 master-1 kubenswrapper[4771]: I1011 11:03:45.971884 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/octavia-api-84f885c68-ttgvk"] Oct 11 11:03:46.451999 master-1 kubenswrapper[4771]: I1011 11:03:46.451877 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" path="/var/lib/kubelet/pods/a67756a2-aa42-4f6f-b27a-57e16f566883/volumes" Oct 11 11:03:47.483731 master-1 kubenswrapper[4771]: I1011 11:03:47.483658 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-image-upload-678599687f-6wlhj"] Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: E1011 11:03:47.484069 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="30a9a1fe-e08c-4112-a7b8-6616d280405e" containerName="octavia-amphora-httpd" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: I1011 11:03:47.484088 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a9a1fe-e08c-4112-a7b8-6616d280405e" containerName="octavia-amphora-httpd" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: E1011 11:03:47.484125 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="octavia-api-provider-agent" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: I1011 11:03:47.484133 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="octavia-api-provider-agent" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: E1011 11:03:47.484147 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="init" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: I1011 11:03:47.484155 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="init" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: E1011 11:03:47.484183 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="30a9a1fe-e08c-4112-a7b8-6616d280405e" containerName="init" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: I1011 11:03:47.484191 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a9a1fe-e08c-4112-a7b8-6616d280405e" containerName="init" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: E1011 11:03:47.484202 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="octavia-api" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: I1011 11:03:47.484211 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="octavia-api" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: I1011 11:03:47.484433 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="30a9a1fe-e08c-4112-a7b8-6616d280405e" containerName="octavia-amphora-httpd" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: I1011 11:03:47.484456 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="octavia-api" Oct 11 11:03:47.484460 master-1 kubenswrapper[4771]: I1011 11:03:47.484467 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a67756a2-aa42-4f6f-b27a-57e16f566883" containerName="octavia-api-provider-agent" Oct 11 11:03:47.485820 master-1 kubenswrapper[4771]: I1011 11:03:47.485772 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-image-upload-678599687f-6wlhj" Oct 11 11:03:47.489769 master-1 kubenswrapper[4771]: I1011 11:03:47.489707 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-config-data" Oct 11 11:03:47.502001 master-1 kubenswrapper[4771]: I1011 11:03:47.501957 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-image-upload-678599687f-6wlhj"] Oct 11 11:03:47.604673 master-1 kubenswrapper[4771]: I1011 11:03:47.604596 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"amphora-image\" (UniqueName: \"kubernetes.io/empty-dir/420f57b5-0516-421c-9eb8-9292d25b970d-amphora-image\") pod \"octavia-image-upload-678599687f-6wlhj\" (UID: \"420f57b5-0516-421c-9eb8-9292d25b970d\") " pod="openstack/octavia-image-upload-678599687f-6wlhj" Oct 11 11:03:47.604962 master-1 kubenswrapper[4771]: I1011 11:03:47.604878 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/420f57b5-0516-421c-9eb8-9292d25b970d-httpd-config\") pod \"octavia-image-upload-678599687f-6wlhj\" (UID: \"420f57b5-0516-421c-9eb8-9292d25b970d\") " pod="openstack/octavia-image-upload-678599687f-6wlhj" Oct 11 11:03:47.707172 master-1 kubenswrapper[4771]: I1011 11:03:47.707099 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/420f57b5-0516-421c-9eb8-9292d25b970d-httpd-config\") pod \"octavia-image-upload-678599687f-6wlhj\" (UID: \"420f57b5-0516-421c-9eb8-9292d25b970d\") " pod="openstack/octavia-image-upload-678599687f-6wlhj" Oct 11 11:03:47.707440 master-1 kubenswrapper[4771]: I1011 11:03:47.707235 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"amphora-image\" (UniqueName: \"kubernetes.io/empty-dir/420f57b5-0516-421c-9eb8-9292d25b970d-amphora-image\") pod \"octavia-image-upload-678599687f-6wlhj\" (UID: \"420f57b5-0516-421c-9eb8-9292d25b970d\") " pod="openstack/octavia-image-upload-678599687f-6wlhj" Oct 11 11:03:47.707943 master-1 kubenswrapper[4771]: I1011 11:03:47.707908 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"amphora-image\" (UniqueName: \"kubernetes.io/empty-dir/420f57b5-0516-421c-9eb8-9292d25b970d-amphora-image\") pod \"octavia-image-upload-678599687f-6wlhj\" (UID: \"420f57b5-0516-421c-9eb8-9292d25b970d\") " pod="openstack/octavia-image-upload-678599687f-6wlhj" Oct 11 11:03:47.716048 master-1 kubenswrapper[4771]: I1011 11:03:47.715983 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"httpd-config\" (UniqueName: \"kubernetes.io/secret/420f57b5-0516-421c-9eb8-9292d25b970d-httpd-config\") pod \"octavia-image-upload-678599687f-6wlhj\" (UID: \"420f57b5-0516-421c-9eb8-9292d25b970d\") " pod="openstack/octavia-image-upload-678599687f-6wlhj" Oct 11 11:03:47.854249 master-1 kubenswrapper[4771]: I1011 11:03:47.854187 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-image-upload-678599687f-6wlhj" Oct 11 11:03:48.280689 master-1 kubenswrapper[4771]: I1011 11:03:48.279183 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-image-upload-678599687f-6wlhj"] Oct 11 11:03:48.284986 master-1 kubenswrapper[4771]: W1011 11:03:48.284932 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod420f57b5_0516_421c_9eb8_9292d25b970d.slice/crio-0d1e950d86bc9f254cc676bdf73db3c7876708e150adca1f5aa8577ebd603f8f WatchSource:0}: Error finding container 0d1e950d86bc9f254cc676bdf73db3c7876708e150adca1f5aa8577ebd603f8f: Status 404 returned error can't find the container with id 0d1e950d86bc9f254cc676bdf73db3c7876708e150adca1f5aa8577ebd603f8f Oct 11 11:03:48.590406 master-1 kubenswrapper[4771]: I1011 11:03:48.590311 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-image-upload-678599687f-6wlhj" event={"ID":"420f57b5-0516-421c-9eb8-9292d25b970d","Type":"ContainerStarted","Data":"0d1e950d86bc9f254cc676bdf73db3c7876708e150adca1f5aa8577ebd603f8f"} Oct 11 11:03:49.605908 master-1 kubenswrapper[4771]: I1011 11:03:49.605796 4771 generic.go:334] "Generic (PLEG): container finished" podID="420f57b5-0516-421c-9eb8-9292d25b970d" containerID="dd07cd4c096ade4f01dd9d10c239a3de8236d4cbc66416bdef796255440f219a" exitCode=0 Oct 11 11:03:49.605908 master-1 kubenswrapper[4771]: I1011 11:03:49.605902 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-image-upload-678599687f-6wlhj" event={"ID":"420f57b5-0516-421c-9eb8-9292d25b970d","Type":"ContainerDied","Data":"dd07cd4c096ade4f01dd9d10c239a3de8236d4cbc66416bdef796255440f219a"} Oct 11 11:03:50.623236 master-1 kubenswrapper[4771]: I1011 11:03:50.623132 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-image-upload-678599687f-6wlhj" event={"ID":"420f57b5-0516-421c-9eb8-9292d25b970d","Type":"ContainerStarted","Data":"89c0833ca2cccadd62ed3e1986975324540ae427e61e10a9a0811c668a54aeb4"} Oct 11 11:03:50.674855 master-1 kubenswrapper[4771]: I1011 11:03:50.674756 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/octavia-image-upload-678599687f-6wlhj" podStartSLOduration=3.221825886 podStartE2EDuration="3.6747289s" podCreationTimestamp="2025-10-11 11:03:47 +0000 UTC" firstStartedPulling="2025-10-11 11:03:48.287933879 +0000 UTC m=+2260.262160350" lastFinishedPulling="2025-10-11 11:03:48.740836923 +0000 UTC m=+2260.715063364" observedRunningTime="2025-10-11 11:03:50.65766053 +0000 UTC m=+2262.631887031" watchObservedRunningTime="2025-10-11 11:03:50.6747289 +0000 UTC m=+2262.648955341" Oct 11 11:03:56.993326 master-1 kubenswrapper[4771]: I1011 11:03:56.993241 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/ceilometer-0" Oct 11 11:04:08.778288 master-1 kubenswrapper[4771]: I1011 11:04:08.778204 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-healthmanager-j6frc"] Oct 11 11:04:08.780638 master-1 kubenswrapper[4771]: I1011 11:04:08.780591 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.785912 master-1 kubenswrapper[4771]: I1011 11:04:08.785867 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-healthmanager-config-data" Oct 11 11:04:08.786106 master-1 kubenswrapper[4771]: I1011 11:04:08.786082 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-certs-secret" Oct 11 11:04:08.786280 master-1 kubenswrapper[4771]: I1011 11:04:08.786237 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-healthmanager-scripts" Oct 11 11:04:08.808042 master-1 kubenswrapper[4771]: I1011 11:04:08.807972 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-healthmanager-j6frc"] Oct 11 11:04:08.843793 master-1 kubenswrapper[4771]: I1011 11:04:08.843705 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"amphora-certs\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-amphora-certs\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.843793 master-1 kubenswrapper[4771]: I1011 11:04:08.843775 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-combined-ca-bundle\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.844221 master-1 kubenswrapper[4771]: I1011 11:04:08.843829 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/c02195a5-59ae-446e-9984-abafa5c03ce5-hm-ports\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.844221 master-1 kubenswrapper[4771]: I1011 11:04:08.843856 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/c02195a5-59ae-446e-9984-abafa5c03ce5-config-data-merged\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.844221 master-1 kubenswrapper[4771]: I1011 11:04:08.844000 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-config-data\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.844221 master-1 kubenswrapper[4771]: I1011 11:04:08.844080 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-scripts\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.945063 master-1 kubenswrapper[4771]: I1011 11:04:08.945008 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-config-data\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.945457 master-1 kubenswrapper[4771]: I1011 11:04:08.945438 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-scripts\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.945574 master-1 kubenswrapper[4771]: I1011 11:04:08.945560 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"amphora-certs\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-amphora-certs\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.945666 master-1 kubenswrapper[4771]: I1011 11:04:08.945651 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-combined-ca-bundle\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.945783 master-1 kubenswrapper[4771]: I1011 11:04:08.945769 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/c02195a5-59ae-446e-9984-abafa5c03ce5-hm-ports\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.945885 master-1 kubenswrapper[4771]: I1011 11:04:08.945873 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/c02195a5-59ae-446e-9984-abafa5c03ce5-config-data-merged\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.946486 master-1 kubenswrapper[4771]: I1011 11:04:08.946470 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/c02195a5-59ae-446e-9984-abafa5c03ce5-config-data-merged\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.948983 master-1 kubenswrapper[4771]: I1011 11:04:08.948923 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/c02195a5-59ae-446e-9984-abafa5c03ce5-hm-ports\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.950672 master-1 kubenswrapper[4771]: I1011 11:04:08.950652 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-config-data\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.951621 master-1 kubenswrapper[4771]: I1011 11:04:08.951599 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"amphora-certs\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-amphora-certs\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.952845 master-1 kubenswrapper[4771]: I1011 11:04:08.952784 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-scripts\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:08.954602 master-1 kubenswrapper[4771]: I1011 11:04:08.954551 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c02195a5-59ae-446e-9984-abafa5c03ce5-combined-ca-bundle\") pod \"octavia-healthmanager-j6frc\" (UID: \"c02195a5-59ae-446e-9984-abafa5c03ce5\") " pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:09.104252 master-1 kubenswrapper[4771]: I1011 11:04:09.104176 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:09.897259 master-1 kubenswrapper[4771]: I1011 11:04:09.897197 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-healthmanager-j6frc"] Oct 11 11:04:10.867185 master-1 kubenswrapper[4771]: I1011 11:04:10.867119 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-healthmanager-j6frc" event={"ID":"c02195a5-59ae-446e-9984-abafa5c03ce5","Type":"ContainerStarted","Data":"f78da3624b576ac119751eab0fd297c9217f5f3b2082190b2dcca93bfd67d131"} Oct 11 11:04:10.867185 master-1 kubenswrapper[4771]: I1011 11:04:10.867186 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-healthmanager-j6frc" event={"ID":"c02195a5-59ae-446e-9984-abafa5c03ce5","Type":"ContainerStarted","Data":"0da54d2dd711becbc625a7d84e560e520bc18693d22425761a185f6ecc5e0bc7"} Oct 11 11:04:11.047804 master-1 kubenswrapper[4771]: I1011 11:04:11.047735 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-housekeeping-hjq8j"] Oct 11 11:04:11.050330 master-1 kubenswrapper[4771]: I1011 11:04:11.050199 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.059669 master-1 kubenswrapper[4771]: I1011 11:04:11.059609 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-housekeeping-scripts" Oct 11 11:04:11.060053 master-1 kubenswrapper[4771]: I1011 11:04:11.059614 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-housekeeping-config-data" Oct 11 11:04:11.067440 master-1 kubenswrapper[4771]: I1011 11:04:11.067253 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-housekeeping-hjq8j"] Oct 11 11:04:11.099464 master-1 kubenswrapper[4771]: I1011 11:04:11.099367 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"amphora-certs\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-amphora-certs\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.099695 master-1 kubenswrapper[4771]: I1011 11:04:11.099499 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/4147847f-c490-4bc6-a2da-298d3a3e188b-config-data-merged\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.099695 master-1 kubenswrapper[4771]: I1011 11:04:11.099550 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-scripts\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.099988 master-1 kubenswrapper[4771]: I1011 11:04:11.099739 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-config-data\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.100053 master-1 kubenswrapper[4771]: I1011 11:04:11.100007 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/4147847f-c490-4bc6-a2da-298d3a3e188b-hm-ports\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.100269 master-1 kubenswrapper[4771]: I1011 11:04:11.100244 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-combined-ca-bundle\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.203653 master-1 kubenswrapper[4771]: I1011 11:04:11.202872 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-combined-ca-bundle\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.203653 master-1 kubenswrapper[4771]: I1011 11:04:11.202984 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"amphora-certs\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-amphora-certs\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.203653 master-1 kubenswrapper[4771]: I1011 11:04:11.203034 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/4147847f-c490-4bc6-a2da-298d3a3e188b-config-data-merged\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.203653 master-1 kubenswrapper[4771]: I1011 11:04:11.203068 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-scripts\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.203653 master-1 kubenswrapper[4771]: I1011 11:04:11.203094 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-config-data\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.203653 master-1 kubenswrapper[4771]: I1011 11:04:11.203145 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/4147847f-c490-4bc6-a2da-298d3a3e188b-hm-ports\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.204618 master-1 kubenswrapper[4771]: I1011 11:04:11.204143 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/4147847f-c490-4bc6-a2da-298d3a3e188b-config-data-merged\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.205272 master-1 kubenswrapper[4771]: I1011 11:04:11.205124 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/4147847f-c490-4bc6-a2da-298d3a3e188b-hm-ports\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.208036 master-1 kubenswrapper[4771]: I1011 11:04:11.207951 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"amphora-certs\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-amphora-certs\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.210192 master-1 kubenswrapper[4771]: I1011 11:04:11.210116 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-config-data\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.211742 master-1 kubenswrapper[4771]: I1011 11:04:11.211203 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-combined-ca-bundle\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.213463 master-1 kubenswrapper[4771]: I1011 11:04:11.213400 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/4147847f-c490-4bc6-a2da-298d3a3e188b-scripts\") pod \"octavia-housekeeping-hjq8j\" (UID: \"4147847f-c490-4bc6-a2da-298d3a3e188b\") " pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:11.376539 master-1 kubenswrapper[4771]: I1011 11:04:11.375331 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:12.849924 master-1 kubenswrapper[4771]: I1011 11:04:12.849701 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/octavia-worker-jwl5z"] Oct 11 11:04:12.854874 master-1 kubenswrapper[4771]: I1011 11:04:12.854796 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:12.860931 master-1 kubenswrapper[4771]: I1011 11:04:12.860867 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-worker-config-data" Oct 11 11:04:12.861139 master-1 kubenswrapper[4771]: I1011 11:04:12.861094 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"octavia-worker-scripts" Oct 11 11:04:12.864276 master-1 kubenswrapper[4771]: I1011 11:04:12.864216 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-worker-jwl5z"] Oct 11 11:04:12.897200 master-1 kubenswrapper[4771]: I1011 11:04:12.897142 4771 generic.go:334] "Generic (PLEG): container finished" podID="c02195a5-59ae-446e-9984-abafa5c03ce5" containerID="f78da3624b576ac119751eab0fd297c9217f5f3b2082190b2dcca93bfd67d131" exitCode=0 Oct 11 11:04:12.897443 master-1 kubenswrapper[4771]: I1011 11:04:12.897209 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-healthmanager-j6frc" event={"ID":"c02195a5-59ae-446e-9984-abafa5c03ce5","Type":"ContainerDied","Data":"f78da3624b576ac119751eab0fd297c9217f5f3b2082190b2dcca93bfd67d131"} Oct 11 11:04:12.954471 master-1 kubenswrapper[4771]: I1011 11:04:12.953830 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/2de0e65e-d3d5-4b50-939a-afc198469db7-hm-ports\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:12.954471 master-1 kubenswrapper[4771]: I1011 11:04:12.953963 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"amphora-certs\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-amphora-certs\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:12.954471 master-1 kubenswrapper[4771]: I1011 11:04:12.953994 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-scripts\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:12.954471 master-1 kubenswrapper[4771]: I1011 11:04:12.954134 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/2de0e65e-d3d5-4b50-939a-afc198469db7-config-data-merged\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:12.954471 master-1 kubenswrapper[4771]: I1011 11:04:12.954247 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-combined-ca-bundle\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:12.954471 master-1 kubenswrapper[4771]: I1011 11:04:12.954345 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-config-data\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.057049 master-1 kubenswrapper[4771]: I1011 11:04:13.056902 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-config-data\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.057489 master-1 kubenswrapper[4771]: I1011 11:04:13.057157 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/2de0e65e-d3d5-4b50-939a-afc198469db7-hm-ports\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.057489 master-1 kubenswrapper[4771]: I1011 11:04:13.057266 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"amphora-certs\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-amphora-certs\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.057489 master-1 kubenswrapper[4771]: I1011 11:04:13.057317 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-scripts\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.057489 master-1 kubenswrapper[4771]: I1011 11:04:13.057350 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/2de0e65e-d3d5-4b50-939a-afc198469db7-config-data-merged\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.057489 master-1 kubenswrapper[4771]: I1011 11:04:13.057407 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-combined-ca-bundle\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.058682 master-1 kubenswrapper[4771]: I1011 11:04:13.058620 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data-merged\" (UniqueName: \"kubernetes.io/empty-dir/2de0e65e-d3d5-4b50-939a-afc198469db7-config-data-merged\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.060251 master-1 kubenswrapper[4771]: I1011 11:04:13.060193 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"hm-ports\" (UniqueName: \"kubernetes.io/configmap/2de0e65e-d3d5-4b50-939a-afc198469db7-hm-ports\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.062190 master-1 kubenswrapper[4771]: I1011 11:04:13.062148 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"scripts\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-scripts\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.063128 master-1 kubenswrapper[4771]: I1011 11:04:13.063081 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"amphora-certs\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-amphora-certs\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.064336 master-1 kubenswrapper[4771]: I1011 11:04:13.064307 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-combined-ca-bundle\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.065232 master-1 kubenswrapper[4771]: I1011 11:04:13.065203 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-data\" (UniqueName: \"kubernetes.io/secret/2de0e65e-d3d5-4b50-939a-afc198469db7-config-data\") pod \"octavia-worker-jwl5z\" (UID: \"2de0e65e-d3d5-4b50-939a-afc198469db7\") " pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.211113 master-1 kubenswrapper[4771]: I1011 11:04:13.210697 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:13.817442 master-1 kubenswrapper[4771]: I1011 11:04:13.814912 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-worker-jwl5z"] Oct 11 11:04:13.818640 master-1 kubenswrapper[4771]: W1011 11:04:13.818589 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de0e65e_d3d5_4b50_939a_afc198469db7.slice/crio-620af9e2516377bf5ba951b3c9e51790ef07e4d8539330389c86481c3c629618 WatchSource:0}: Error finding container 620af9e2516377bf5ba951b3c9e51790ef07e4d8539330389c86481c3c629618: Status 404 returned error can't find the container with id 620af9e2516377bf5ba951b3c9e51790ef07e4d8539330389c86481c3c629618 Oct 11 11:04:13.908381 master-1 kubenswrapper[4771]: I1011 11:04:13.908270 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-worker-jwl5z" event={"ID":"2de0e65e-d3d5-4b50-939a-afc198469db7","Type":"ContainerStarted","Data":"620af9e2516377bf5ba951b3c9e51790ef07e4d8539330389c86481c3c629618"} Oct 11 11:04:13.910495 master-1 kubenswrapper[4771]: I1011 11:04:13.910346 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-healthmanager-j6frc" event={"ID":"c02195a5-59ae-446e-9984-abafa5c03ce5","Type":"ContainerStarted","Data":"8187c3e5dc58271c299604a12745842c20da91d99b4e5d0c317b8819d82e8696"} Oct 11 11:04:13.910905 master-1 kubenswrapper[4771]: I1011 11:04:13.910836 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:13.952686 master-1 kubenswrapper[4771]: I1011 11:04:13.952595 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/octavia-healthmanager-j6frc" podStartSLOduration=5.95257079 podStartE2EDuration="5.95257079s" podCreationTimestamp="2025-10-11 11:04:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 11:04:13.950982945 +0000 UTC m=+2285.925209396" watchObservedRunningTime="2025-10-11 11:04:13.95257079 +0000 UTC m=+2285.926797231" Oct 11 11:04:14.515379 master-1 kubenswrapper[4771]: I1011 11:04:14.515018 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/octavia-housekeeping-hjq8j"] Oct 11 11:04:14.924374 master-1 kubenswrapper[4771]: I1011 11:04:14.923986 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-housekeeping-hjq8j" event={"ID":"4147847f-c490-4bc6-a2da-298d3a3e188b","Type":"ContainerStarted","Data":"3a732fe908f149e77768cda1e245a5bbde89d124df44b7c8f34da2dbc422e246"} Oct 11 11:04:16.951992 master-1 kubenswrapper[4771]: I1011 11:04:16.951911 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-housekeeping-hjq8j" event={"ID":"4147847f-c490-4bc6-a2da-298d3a3e188b","Type":"ContainerStarted","Data":"785f888f163f9a4ad23b11870a89d933866aaf82fd9811e43f9ca7e883e62d16"} Oct 11 11:04:17.962562 master-1 kubenswrapper[4771]: I1011 11:04:17.962500 4771 generic.go:334] "Generic (PLEG): container finished" podID="2de0e65e-d3d5-4b50-939a-afc198469db7" containerID="775bdd8e43e99152441a14ec4179896b210af621ae7682d1c65a8fc5e6b85b65" exitCode=0 Oct 11 11:04:17.963395 master-1 kubenswrapper[4771]: I1011 11:04:17.962862 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-worker-jwl5z" event={"ID":"2de0e65e-d3d5-4b50-939a-afc198469db7","Type":"ContainerDied","Data":"775bdd8e43e99152441a14ec4179896b210af621ae7682d1c65a8fc5e6b85b65"} Oct 11 11:04:17.966743 master-1 kubenswrapper[4771]: I1011 11:04:17.966715 4771 generic.go:334] "Generic (PLEG): container finished" podID="4147847f-c490-4bc6-a2da-298d3a3e188b" containerID="785f888f163f9a4ad23b11870a89d933866aaf82fd9811e43f9ca7e883e62d16" exitCode=0 Oct 11 11:04:17.966875 master-1 kubenswrapper[4771]: I1011 11:04:17.966756 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-housekeeping-hjq8j" event={"ID":"4147847f-c490-4bc6-a2da-298d3a3e188b","Type":"ContainerDied","Data":"785f888f163f9a4ad23b11870a89d933866aaf82fd9811e43f9ca7e883e62d16"} Oct 11 11:04:19.000080 master-1 kubenswrapper[4771]: I1011 11:04:19.000004 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-housekeeping-hjq8j" event={"ID":"4147847f-c490-4bc6-a2da-298d3a3e188b","Type":"ContainerStarted","Data":"afe19be260ccae21154710d06bcbde1d2b07e18920105bb8499fcf325b4e6df4"} Oct 11 11:04:19.003475 master-1 kubenswrapper[4771]: I1011 11:04:19.003412 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/octavia-worker-jwl5z" event={"ID":"2de0e65e-d3d5-4b50-939a-afc198469db7","Type":"ContainerStarted","Data":"2ccf70b97f25e460b07e34a6a246840a964d663547005c15c82774fd6b717cf5"} Oct 11 11:04:19.004278 master-1 kubenswrapper[4771]: I1011 11:04:19.004222 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:19.045556 master-1 kubenswrapper[4771]: I1011 11:04:19.045417 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/octavia-housekeeping-hjq8j" podStartSLOduration=6.692584599 podStartE2EDuration="8.045392016s" podCreationTimestamp="2025-10-11 11:04:11 +0000 UTC" firstStartedPulling="2025-10-11 11:04:14.525178289 +0000 UTC m=+2286.499404720" lastFinishedPulling="2025-10-11 11:04:15.877985696 +0000 UTC m=+2287.852212137" observedRunningTime="2025-10-11 11:04:19.03574455 +0000 UTC m=+2291.009971011" watchObservedRunningTime="2025-10-11 11:04:19.045392016 +0000 UTC m=+2291.019618477" Oct 11 11:04:19.067681 master-1 kubenswrapper[4771]: I1011 11:04:19.067052 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/octavia-worker-jwl5z" podStartSLOduration=4.46827366 podStartE2EDuration="7.067024215s" podCreationTimestamp="2025-10-11 11:04:12 +0000 UTC" firstStartedPulling="2025-10-11 11:04:13.822881951 +0000 UTC m=+2285.797108392" lastFinishedPulling="2025-10-11 11:04:16.421632506 +0000 UTC m=+2288.395858947" observedRunningTime="2025-10-11 11:04:19.065470481 +0000 UTC m=+2291.039696952" watchObservedRunningTime="2025-10-11 11:04:19.067024215 +0000 UTC m=+2291.041250656" Oct 11 11:04:20.013387 master-1 kubenswrapper[4771]: I1011 11:04:20.013289 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:23.777002 master-1 kubenswrapper[4771]: I1011 11:04:23.776913 4771 scope.go:117] "RemoveContainer" containerID="497e015434504b4db642357797a1c623d7b35238dcc0952d89c6a79885be7010" Oct 11 11:04:24.163753 master-1 kubenswrapper[4771]: I1011 11:04:24.163638 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/octavia-healthmanager-j6frc" Oct 11 11:04:26.407520 master-1 kubenswrapper[4771]: I1011 11:04:26.407453 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/octavia-housekeeping-hjq8j" Oct 11 11:04:28.241119 master-1 kubenswrapper[4771]: I1011 11:04:28.241045 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/octavia-worker-jwl5z" Oct 11 11:04:57.758911 master-1 kubenswrapper[4771]: I1011 11:04:57.758602 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-6cb9b8c955-b5qwc"] Oct 11 11:04:57.760789 master-1 kubenswrapper[4771]: I1011 11:04:57.760712 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" podUID="396249ad-10d3-48d9-ba43-46df789198c9" containerName="dnsmasq-dns" containerID="cri-o://6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760" gracePeriod=10 Oct 11 11:04:57.816647 master-1 kubenswrapper[4771]: I1011 11:04:57.816499 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-85b986dbf-gg6pg"] Oct 11 11:04:57.823555 master-1 kubenswrapper[4771]: I1011 11:04:57.823467 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:57.831326 master-1 kubenswrapper[4771]: I1011 11:04:57.831275 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"networkers" Oct 11 11:04:57.831701 master-1 kubenswrapper[4771]: I1011 11:04:57.831582 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-85b986dbf-gg6pg"] Oct 11 11:04:57.947043 master-1 kubenswrapper[4771]: I1011 11:04:57.946944 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqqv\" (UniqueName: \"kubernetes.io/projected/ba5730cb-34e6-498e-995a-20912130efe3-kube-api-access-8sqqv\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:57.947472 master-1 kubenswrapper[4771]: I1011 11:04:57.947216 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-config\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:57.947472 master-1 kubenswrapper[4771]: I1011 11:04:57.947263 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-swift-storage-0\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:57.947472 master-1 kubenswrapper[4771]: I1011 11:04:57.947295 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-svc\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:57.947472 master-1 kubenswrapper[4771]: I1011 11:04:57.947317 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-sb\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:57.947472 master-1 kubenswrapper[4771]: I1011 11:04:57.947457 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-nb\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:57.948341 master-1 kubenswrapper[4771]: I1011 11:04:57.947520 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-networkers\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.013007 master-1 kubenswrapper[4771]: I1011 11:04:57.993092 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-85b986dbf-gg6pg"] Oct 11 11:04:58.013007 master-1 kubenswrapper[4771]: E1011 11:04:57.993663 4771 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[config dns-svc dns-swift-storage-0 kube-api-access-8sqqv networkers ovsdbserver-nb ovsdbserver-sb], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" podUID="ba5730cb-34e6-498e-995a-20912130efe3" Oct 11 11:04:58.048986 master-1 kubenswrapper[4771]: I1011 11:04:58.048889 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqqv\" (UniqueName: \"kubernetes.io/projected/ba5730cb-34e6-498e-995a-20912130efe3-kube-api-access-8sqqv\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.048986 master-1 kubenswrapper[4771]: I1011 11:04:58.048998 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-config\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.049696 master-1 kubenswrapper[4771]: I1011 11:04:58.049021 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-swift-storage-0\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.049696 master-1 kubenswrapper[4771]: I1011 11:04:58.049049 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-svc\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.049696 master-1 kubenswrapper[4771]: I1011 11:04:58.049075 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-sb\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.049696 master-1 kubenswrapper[4771]: I1011 11:04:58.049136 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-nb\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.049696 master-1 kubenswrapper[4771]: I1011 11:04:58.049195 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-networkers\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.050299 master-1 kubenswrapper[4771]: I1011 11:04:58.050241 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-swift-storage-0\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.050404 master-1 kubenswrapper[4771]: I1011 11:04:58.050377 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-svc\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.050976 master-1 kubenswrapper[4771]: I1011 11:04:58.050917 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-sb\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.051157 master-1 kubenswrapper[4771]: I1011 11:04:58.051027 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-nb\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.051157 master-1 kubenswrapper[4771]: I1011 11:04:58.050994 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-config\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.051688 master-1 kubenswrapper[4771]: I1011 11:04:58.051654 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-networkers\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.080293 master-1 kubenswrapper[4771]: I1011 11:04:58.080203 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqqv\" (UniqueName: \"kubernetes.io/projected/ba5730cb-34e6-498e-995a-20912130efe3-kube-api-access-8sqqv\") pod \"dnsmasq-dns-85b986dbf-gg6pg\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.405163 master-1 kubenswrapper[4771]: I1011 11:04:58.405078 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 11:04:58.471832 master-1 kubenswrapper[4771]: I1011 11:04:58.471752 4771 generic.go:334] "Generic (PLEG): container finished" podID="396249ad-10d3-48d9-ba43-46df789198c9" containerID="6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760" exitCode=0 Oct 11 11:04:58.471832 master-1 kubenswrapper[4771]: I1011 11:04:58.471823 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" Oct 11 11:04:58.472964 master-1 kubenswrapper[4771]: I1011 11:04:58.471856 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.472964 master-1 kubenswrapper[4771]: I1011 11:04:58.471877 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" event={"ID":"396249ad-10d3-48d9-ba43-46df789198c9","Type":"ContainerDied","Data":"6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760"} Oct 11 11:04:58.472964 master-1 kubenswrapper[4771]: I1011 11:04:58.471968 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-6cb9b8c955-b5qwc" event={"ID":"396249ad-10d3-48d9-ba43-46df789198c9","Type":"ContainerDied","Data":"a14cd6677525c65737d0849bb25554909c4ebb8c2b5761120df0ab99b361a3df"} Oct 11 11:04:58.472964 master-1 kubenswrapper[4771]: I1011 11:04:58.471993 4771 scope.go:117] "RemoveContainer" containerID="6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760" Oct 11 11:04:58.481064 master-1 kubenswrapper[4771]: I1011 11:04:58.481001 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:58.499801 master-1 kubenswrapper[4771]: I1011 11:04:58.499723 4771 scope.go:117] "RemoveContainer" containerID="56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2" Oct 11 11:04:58.528005 master-1 kubenswrapper[4771]: I1011 11:04:58.527928 4771 scope.go:117] "RemoveContainer" containerID="6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760" Oct 11 11:04:58.528742 master-1 kubenswrapper[4771]: E1011 11:04:58.528693 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760\": container with ID starting with 6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760 not found: ID does not exist" containerID="6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760" Oct 11 11:04:58.528825 master-1 kubenswrapper[4771]: I1011 11:04:58.528739 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760"} err="failed to get container status \"6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760\": rpc error: code = NotFound desc = could not find container \"6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760\": container with ID starting with 6510e0b21de1c8b7cb314f294d9600ddbe32a0fc8c522f5123cca3a802773760 not found: ID does not exist" Oct 11 11:04:58.528825 master-1 kubenswrapper[4771]: I1011 11:04:58.528771 4771 scope.go:117] "RemoveContainer" containerID="56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2" Oct 11 11:04:58.529284 master-1 kubenswrapper[4771]: E1011 11:04:58.529242 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2\": container with ID starting with 56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2 not found: ID does not exist" containerID="56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2" Oct 11 11:04:58.529284 master-1 kubenswrapper[4771]: I1011 11:04:58.529269 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2"} err="failed to get container status \"56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2\": rpc error: code = NotFound desc = could not find container \"56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2\": container with ID starting with 56676cd3860cb8b15840c440ca73c01a49ae818c01bccb1e7e3f799e36cf02d2 not found: ID does not exist" Oct 11 11:04:58.563776 master-1 kubenswrapper[4771]: I1011 11:04:58.563695 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-config\") pod \"ba5730cb-34e6-498e-995a-20912130efe3\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " Oct 11 11:04:58.563953 master-1 kubenswrapper[4771]: I1011 11:04:58.563782 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-config\") pod \"396249ad-10d3-48d9-ba43-46df789198c9\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " Oct 11 11:04:58.563953 master-1 kubenswrapper[4771]: I1011 11:04:58.563868 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-nb\") pod \"396249ad-10d3-48d9-ba43-46df789198c9\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " Oct 11 11:04:58.564052 master-1 kubenswrapper[4771]: I1011 11:04:58.563984 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-nb\") pod \"ba5730cb-34e6-498e-995a-20912130efe3\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " Oct 11 11:04:58.564667 master-1 kubenswrapper[4771]: I1011 11:04:58.564607 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-config" (OuterVolumeSpecName: "config") pod "ba5730cb-34e6-498e-995a-20912130efe3" (UID: "ba5730cb-34e6-498e-995a-20912130efe3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.564756 master-1 kubenswrapper[4771]: I1011 11:04:58.564654 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-nb" (OuterVolumeSpecName: "ovsdbserver-nb") pod "ba5730cb-34e6-498e-995a-20912130efe3" (UID: "ba5730cb-34e6-498e-995a-20912130efe3"). InnerVolumeSpecName "ovsdbserver-nb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.564847 master-1 kubenswrapper[4771]: I1011 11:04:58.564811 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-sb\") pod \"ba5730cb-34e6-498e-995a-20912130efe3\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " Oct 11 11:04:58.564941 master-1 kubenswrapper[4771]: I1011 11:04:58.564911 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-sb\") pod \"396249ad-10d3-48d9-ba43-46df789198c9\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " Oct 11 11:04:58.565020 master-1 kubenswrapper[4771]: I1011 11:04:58.564973 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-swift-storage-0\") pod \"ba5730cb-34e6-498e-995a-20912130efe3\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " Oct 11 11:04:58.565351 master-1 kubenswrapper[4771]: I1011 11:04:58.565324 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-networkers\") pod \"ba5730cb-34e6-498e-995a-20912130efe3\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " Oct 11 11:04:58.565488 master-1 kubenswrapper[4771]: I1011 11:04:58.565346 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-sb" (OuterVolumeSpecName: "ovsdbserver-sb") pod "ba5730cb-34e6-498e-995a-20912130efe3" (UID: "ba5730cb-34e6-498e-995a-20912130efe3"). InnerVolumeSpecName "ovsdbserver-sb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.565488 master-1 kubenswrapper[4771]: I1011 11:04:58.565392 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-svc\") pod \"ba5730cb-34e6-498e-995a-20912130efe3\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " Oct 11 11:04:58.565488 master-1 kubenswrapper[4771]: I1011 11:04:58.565436 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sqqv\" (UniqueName: \"kubernetes.io/projected/ba5730cb-34e6-498e-995a-20912130efe3-kube-api-access-8sqqv\") pod \"ba5730cb-34e6-498e-995a-20912130efe3\" (UID: \"ba5730cb-34e6-498e-995a-20912130efe3\") " Oct 11 11:04:58.565488 master-1 kubenswrapper[4771]: I1011 11:04:58.565473 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-svc\") pod \"396249ad-10d3-48d9-ba43-46df789198c9\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " Oct 11 11:04:58.566491 master-1 kubenswrapper[4771]: I1011 11:04:58.565875 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-swift-storage-0" (OuterVolumeSpecName: "dns-swift-storage-0") pod "ba5730cb-34e6-498e-995a-20912130efe3" (UID: "ba5730cb-34e6-498e-995a-20912130efe3"). InnerVolumeSpecName "dns-swift-storage-0". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.566491 master-1 kubenswrapper[4771]: I1011 11:04:58.566044 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/396249ad-10d3-48d9-ba43-46df789198c9-kube-api-access-897k7\") pod \"396249ad-10d3-48d9-ba43-46df789198c9\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " Oct 11 11:04:58.566491 master-1 kubenswrapper[4771]: I1011 11:04:58.566085 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-swift-storage-0\") pod \"396249ad-10d3-48d9-ba43-46df789198c9\" (UID: \"396249ad-10d3-48d9-ba43-46df789198c9\") " Oct 11 11:04:58.566491 master-1 kubenswrapper[4771]: I1011 11:04:58.566324 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-networkers" (OuterVolumeSpecName: "networkers") pod "ba5730cb-34e6-498e-995a-20912130efe3" (UID: "ba5730cb-34e6-498e-995a-20912130efe3"). InnerVolumeSpecName "networkers". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.566491 master-1 kubenswrapper[4771]: I1011 11:04:58.566446 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "ba5730cb-34e6-498e-995a-20912130efe3" (UID: "ba5730cb-34e6-498e-995a-20912130efe3"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.567431 master-1 kubenswrapper[4771]: I1011 11:04:58.567396 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-nb\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.567431 master-1 kubenswrapper[4771]: I1011 11:04:58.567428 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-ovsdbserver-sb\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.567604 master-1 kubenswrapper[4771]: I1011 11:04:58.567469 4771 reconciler_common.go:293] "Volume detached for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-swift-storage-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.567604 master-1 kubenswrapper[4771]: I1011 11:04:58.567487 4771 reconciler_common.go:293] "Volume detached for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-networkers\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.567604 master-1 kubenswrapper[4771]: I1011 11:04:58.567501 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.567604 master-1 kubenswrapper[4771]: I1011 11:04:58.567513 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba5730cb-34e6-498e-995a-20912130efe3-config\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.568634 master-1 kubenswrapper[4771]: I1011 11:04:58.568579 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396249ad-10d3-48d9-ba43-46df789198c9-kube-api-access-897k7" (OuterVolumeSpecName: "kube-api-access-897k7") pod "396249ad-10d3-48d9-ba43-46df789198c9" (UID: "396249ad-10d3-48d9-ba43-46df789198c9"). InnerVolumeSpecName "kube-api-access-897k7". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:04:58.570583 master-1 kubenswrapper[4771]: I1011 11:04:58.570525 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5730cb-34e6-498e-995a-20912130efe3-kube-api-access-8sqqv" (OuterVolumeSpecName: "kube-api-access-8sqqv") pod "ba5730cb-34e6-498e-995a-20912130efe3" (UID: "ba5730cb-34e6-498e-995a-20912130efe3"). InnerVolumeSpecName "kube-api-access-8sqqv". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:04:58.604259 master-1 kubenswrapper[4771]: I1011 11:04:58.604112 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-sb" (OuterVolumeSpecName: "ovsdbserver-sb") pod "396249ad-10d3-48d9-ba43-46df789198c9" (UID: "396249ad-10d3-48d9-ba43-46df789198c9"). InnerVolumeSpecName "ovsdbserver-sb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.615672 master-1 kubenswrapper[4771]: I1011 11:04:58.615602 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-nb" (OuterVolumeSpecName: "ovsdbserver-nb") pod "396249ad-10d3-48d9-ba43-46df789198c9" (UID: "396249ad-10d3-48d9-ba43-46df789198c9"). InnerVolumeSpecName "ovsdbserver-nb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.620882 master-1 kubenswrapper[4771]: I1011 11:04:58.620817 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-config" (OuterVolumeSpecName: "config") pod "396249ad-10d3-48d9-ba43-46df789198c9" (UID: "396249ad-10d3-48d9-ba43-46df789198c9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.624534 master-1 kubenswrapper[4771]: I1011 11:04:58.624474 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "396249ad-10d3-48d9-ba43-46df789198c9" (UID: "396249ad-10d3-48d9-ba43-46df789198c9"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.634202 master-1 kubenswrapper[4771]: I1011 11:04:58.634151 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-swift-storage-0" (OuterVolumeSpecName: "dns-swift-storage-0") pod "396249ad-10d3-48d9-ba43-46df789198c9" (UID: "396249ad-10d3-48d9-ba43-46df789198c9"). InnerVolumeSpecName "dns-swift-storage-0". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:04:58.669652 master-1 kubenswrapper[4771]: I1011 11:04:58.669606 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-sb\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.670638 master-1 kubenswrapper[4771]: I1011 11:04:58.669874 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-8sqqv\" (UniqueName: \"kubernetes.io/projected/ba5730cb-34e6-498e-995a-20912130efe3-kube-api-access-8sqqv\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.670638 master-1 kubenswrapper[4771]: I1011 11:04:58.669894 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.670638 master-1 kubenswrapper[4771]: I1011 11:04:58.669908 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/396249ad-10d3-48d9-ba43-46df789198c9-kube-api-access-897k7\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.670638 master-1 kubenswrapper[4771]: I1011 11:04:58.669920 4771 reconciler_common.go:293] "Volume detached for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-dns-swift-storage-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.670638 master-1 kubenswrapper[4771]: I1011 11:04:58.669934 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-config\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.670638 master-1 kubenswrapper[4771]: I1011 11:04:58.669946 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/396249ad-10d3-48d9-ba43-46df789198c9-ovsdbserver-nb\") on node \"master-1\" DevicePath \"\"" Oct 11 11:04:58.876926 master-1 kubenswrapper[4771]: I1011 11:04:58.876747 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-6cb9b8c955-b5qwc"] Oct 11 11:04:58.891413 master-1 kubenswrapper[4771]: I1011 11:04:58.891321 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-6cb9b8c955-b5qwc"] Oct 11 11:04:59.481198 master-1 kubenswrapper[4771]: I1011 11:04:59.481149 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-85b986dbf-gg6pg" Oct 11 11:04:59.547592 master-1 kubenswrapper[4771]: I1011 11:04:59.547456 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-85b986dbf-gg6pg"] Oct 11 11:04:59.560643 master-1 kubenswrapper[4771]: I1011 11:04:59.560605 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-85b986dbf-gg6pg"] Oct 11 11:05:00.456514 master-1 kubenswrapper[4771]: I1011 11:05:00.456428 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396249ad-10d3-48d9-ba43-46df789198c9" path="/var/lib/kubelet/pods/396249ad-10d3-48d9-ba43-46df789198c9/volumes" Oct 11 11:05:00.458428 master-1 kubenswrapper[4771]: I1011 11:05:00.458379 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba5730cb-34e6-498e-995a-20912130efe3" path="/var/lib/kubelet/pods/ba5730cb-34e6-498e-995a-20912130efe3/volumes" Oct 11 11:05:08.529793 master-1 kubenswrapper[4771]: I1011 11:05:08.529683 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-59dd57778c-jshg9"] Oct 11 11:05:08.530811 master-1 kubenswrapper[4771]: E1011 11:05:08.530198 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="396249ad-10d3-48d9-ba43-46df789198c9" containerName="init" Oct 11 11:05:08.530811 master-1 kubenswrapper[4771]: I1011 11:05:08.530215 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="396249ad-10d3-48d9-ba43-46df789198c9" containerName="init" Oct 11 11:05:08.530811 master-1 kubenswrapper[4771]: E1011 11:05:08.530238 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="396249ad-10d3-48d9-ba43-46df789198c9" containerName="dnsmasq-dns" Oct 11 11:05:08.530811 master-1 kubenswrapper[4771]: I1011 11:05:08.530246 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="396249ad-10d3-48d9-ba43-46df789198c9" containerName="dnsmasq-dns" Oct 11 11:05:08.530811 master-1 kubenswrapper[4771]: I1011 11:05:08.530497 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="396249ad-10d3-48d9-ba43-46df789198c9" containerName="dnsmasq-dns" Oct 11 11:05:08.531756 master-1 kubenswrapper[4771]: I1011 11:05:08.531713 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.556080 master-1 kubenswrapper[4771]: I1011 11:05:08.539114 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"dns-swift-storage-0" Oct 11 11:05:08.556080 master-1 kubenswrapper[4771]: I1011 11:05:08.539216 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovsdbserver-nb" Oct 11 11:05:08.556080 master-1 kubenswrapper[4771]: I1011 11:05:08.539384 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"networkers" Oct 11 11:05:08.557062 master-1 kubenswrapper[4771]: I1011 11:05:08.557000 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"dns" Oct 11 11:05:08.557431 master-1 kubenswrapper[4771]: I1011 11:05:08.557406 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovsdbserver-sb" Oct 11 11:05:08.557522 master-1 kubenswrapper[4771]: I1011 11:05:08.557459 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"dns-svc" Oct 11 11:05:08.600120 master-1 kubenswrapper[4771]: I1011 11:05:08.600038 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-59dd57778c-jshg9"] Oct 11 11:05:08.649196 master-1 kubenswrapper[4771]: I1011 11:05:08.649121 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-swift-storage-0\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.649196 master-1 kubenswrapper[4771]: I1011 11:05:08.649225 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-nb\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.649572 master-1 kubenswrapper[4771]: I1011 11:05:08.649317 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-networkers\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.649572 master-1 kubenswrapper[4771]: I1011 11:05:08.649374 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-config\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.649572 master-1 kubenswrapper[4771]: I1011 11:05:08.649432 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-svc\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.649572 master-1 kubenswrapper[4771]: I1011 11:05:08.649500 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwf6\" (UniqueName: \"kubernetes.io/projected/460e6e8f-ccc4-4952-934c-1d3229573074-kube-api-access-smwf6\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.649730 master-1 kubenswrapper[4771]: I1011 11:05:08.649589 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-sb\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.752836 master-1 kubenswrapper[4771]: I1011 11:05:08.752634 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-sb\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.752836 master-1 kubenswrapper[4771]: I1011 11:05:08.752764 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-sb\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.753185 master-1 kubenswrapper[4771]: I1011 11:05:08.752860 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-swift-storage-0\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.753185 master-1 kubenswrapper[4771]: I1011 11:05:08.752916 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-nb\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.753185 master-1 kubenswrapper[4771]: I1011 11:05:08.752952 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-networkers\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.753864 master-1 kubenswrapper[4771]: I1011 11:05:08.753813 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-nb\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.753968 master-1 kubenswrapper[4771]: I1011 11:05:08.753903 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-config\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.754721 master-1 kubenswrapper[4771]: I1011 11:05:08.754677 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-svc\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.754864 master-1 kubenswrapper[4771]: I1011 11:05:08.754818 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-smwf6\" (UniqueName: \"kubernetes.io/projected/460e6e8f-ccc4-4952-934c-1d3229573074-kube-api-access-smwf6\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.755015 master-1 kubenswrapper[4771]: I1011 11:05:08.754563 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-config\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.755015 master-1 kubenswrapper[4771]: I1011 11:05:08.754492 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-networkers\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.755693 master-1 kubenswrapper[4771]: I1011 11:05:08.755650 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-svc\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.755693 master-1 kubenswrapper[4771]: I1011 11:05:08.754542 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-swift-storage-0\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.782006 master-1 kubenswrapper[4771]: I1011 11:05:08.781918 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwf6\" (UniqueName: \"kubernetes.io/projected/460e6e8f-ccc4-4952-934c-1d3229573074-kube-api-access-smwf6\") pod \"dnsmasq-dns-59dd57778c-jshg9\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:08.878177 master-1 kubenswrapper[4771]: I1011 11:05:08.878082 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:09.351988 master-1 kubenswrapper[4771]: I1011 11:05:09.351918 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-59dd57778c-jshg9"] Oct 11 11:05:09.623454 master-1 kubenswrapper[4771]: I1011 11:05:09.623232 4771 generic.go:334] "Generic (PLEG): container finished" podID="460e6e8f-ccc4-4952-934c-1d3229573074" containerID="4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f" exitCode=0 Oct 11 11:05:09.623454 master-1 kubenswrapper[4771]: I1011 11:05:09.623336 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" event={"ID":"460e6e8f-ccc4-4952-934c-1d3229573074","Type":"ContainerDied","Data":"4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f"} Oct 11 11:05:09.623454 master-1 kubenswrapper[4771]: I1011 11:05:09.623416 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" event={"ID":"460e6e8f-ccc4-4952-934c-1d3229573074","Type":"ContainerStarted","Data":"e322bd0a72d963bfb151ddd53adda74fdeefdc268230c1597845f51e0682ee69"} Oct 11 11:05:10.639459 master-1 kubenswrapper[4771]: I1011 11:05:10.639240 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" event={"ID":"460e6e8f-ccc4-4952-934c-1d3229573074","Type":"ContainerStarted","Data":"47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47"} Oct 11 11:05:10.641165 master-1 kubenswrapper[4771]: I1011 11:05:10.640100 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:10.675730 master-1 kubenswrapper[4771]: I1011 11:05:10.675631 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" podStartSLOduration=2.675602021 podStartE2EDuration="2.675602021s" podCreationTimestamp="2025-10-11 11:05:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 11:05:10.671989668 +0000 UTC m=+2342.646216159" watchObservedRunningTime="2025-10-11 11:05:10.675602021 +0000 UTC m=+2342.649828462" Oct 11 11:05:18.879672 master-1 kubenswrapper[4771]: I1011 11:05:18.879590 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:30.005895 master-1 kubenswrapper[4771]: I1011 11:05:30.005783 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-59dd57778c-jshg9"] Oct 11 11:05:30.008795 master-1 kubenswrapper[4771]: I1011 11:05:30.006189 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" podUID="460e6e8f-ccc4-4952-934c-1d3229573074" containerName="dnsmasq-dns" containerID="cri-o://47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47" gracePeriod=10 Oct 11 11:05:30.057081 master-1 kubenswrapper[4771]: I1011 11:05:30.056995 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/dnsmasq-dns-f984c5fd9-vj7wp"] Oct 11 11:05:30.064142 master-1 kubenswrapper[4771]: I1011 11:05:30.061937 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.066296 master-1 kubenswrapper[4771]: I1011 11:05:30.066218 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"edpm" Oct 11 11:05:30.075413 master-1 kubenswrapper[4771]: I1011 11:05:30.075308 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-f984c5fd9-vj7wp"] Oct 11 11:05:30.259761 master-1 kubenswrapper[4771]: I1011 11:05:30.259493 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-ovsdbserver-sb\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.260115 master-1 kubenswrapper[4771]: I1011 11:05:30.259768 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-networkers\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.260115 master-1 kubenswrapper[4771]: I1011 11:05:30.259882 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-ovsdbserver-nb\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.260115 master-1 kubenswrapper[4771]: I1011 11:05:30.259920 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-dns-swift-storage-0\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.260273 master-1 kubenswrapper[4771]: I1011 11:05:30.260173 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczc9\" (UniqueName: \"kubernetes.io/projected/73899937-c48a-4a79-9bc7-c5f4987908c3-kube-api-access-hczc9\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.260273 master-1 kubenswrapper[4771]: I1011 11:05:30.260213 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-config\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.260273 master-1 kubenswrapper[4771]: I1011 11:05:30.260247 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-dns-svc\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.260679 master-1 kubenswrapper[4771]: I1011 11:05:30.260438 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"edpm\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-edpm\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.364427 master-1 kubenswrapper[4771]: I1011 11:05:30.364232 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-ovsdbserver-sb\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.364811 master-1 kubenswrapper[4771]: I1011 11:05:30.364466 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-networkers\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.364811 master-1 kubenswrapper[4771]: I1011 11:05:30.364553 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-ovsdbserver-nb\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.364811 master-1 kubenswrapper[4771]: I1011 11:05:30.364603 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-dns-swift-storage-0\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.364811 master-1 kubenswrapper[4771]: I1011 11:05:30.364702 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hczc9\" (UniqueName: \"kubernetes.io/projected/73899937-c48a-4a79-9bc7-c5f4987908c3-kube-api-access-hczc9\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.364811 master-1 kubenswrapper[4771]: I1011 11:05:30.364740 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-config\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.364811 master-1 kubenswrapper[4771]: I1011 11:05:30.364778 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-dns-svc\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.365034 master-1 kubenswrapper[4771]: I1011 11:05:30.364830 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"edpm\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-edpm\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.367636 master-1 kubenswrapper[4771]: I1011 11:05:30.367578 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"edpm\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-edpm\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.369629 master-1 kubenswrapper[4771]: I1011 11:05:30.369566 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-ovsdbserver-sb\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.371134 master-1 kubenswrapper[4771]: I1011 11:05:30.371079 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-networkers\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.373477 master-1 kubenswrapper[4771]: I1011 11:05:30.373390 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-dns-swift-storage-0\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.373955 master-1 kubenswrapper[4771]: I1011 11:05:30.373844 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-ovsdbserver-nb\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.374273 master-1 kubenswrapper[4771]: I1011 11:05:30.374240 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-config\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.377347 master-1 kubenswrapper[4771]: I1011 11:05:30.377280 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/73899937-c48a-4a79-9bc7-c5f4987908c3-dns-svc\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.408852 master-1 kubenswrapper[4771]: I1011 11:05:30.408786 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczc9\" (UniqueName: \"kubernetes.io/projected/73899937-c48a-4a79-9bc7-c5f4987908c3-kube-api-access-hczc9\") pod \"dnsmasq-dns-f984c5fd9-vj7wp\" (UID: \"73899937-c48a-4a79-9bc7-c5f4987908c3\") " pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.461515 master-1 kubenswrapper[4771]: I1011 11:05:30.461448 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:30.564176 master-1 kubenswrapper[4771]: I1011 11:05:30.563042 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:30.672922 master-1 kubenswrapper[4771]: I1011 11:05:30.672837 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-nb\") pod \"460e6e8f-ccc4-4952-934c-1d3229573074\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " Oct 11 11:05:30.673142 master-1 kubenswrapper[4771]: I1011 11:05:30.672982 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smwf6\" (UniqueName: \"kubernetes.io/projected/460e6e8f-ccc4-4952-934c-1d3229573074-kube-api-access-smwf6\") pod \"460e6e8f-ccc4-4952-934c-1d3229573074\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " Oct 11 11:05:30.673142 master-1 kubenswrapper[4771]: I1011 11:05:30.673127 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-sb\") pod \"460e6e8f-ccc4-4952-934c-1d3229573074\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " Oct 11 11:05:30.673315 master-1 kubenswrapper[4771]: I1011 11:05:30.673283 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-config\") pod \"460e6e8f-ccc4-4952-934c-1d3229573074\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " Oct 11 11:05:30.673391 master-1 kubenswrapper[4771]: I1011 11:05:30.673313 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-svc\") pod \"460e6e8f-ccc4-4952-934c-1d3229573074\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " Oct 11 11:05:30.673391 master-1 kubenswrapper[4771]: I1011 11:05:30.673359 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-swift-storage-0\") pod \"460e6e8f-ccc4-4952-934c-1d3229573074\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " Oct 11 11:05:30.674097 master-1 kubenswrapper[4771]: I1011 11:05:30.674068 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-networkers\") pod \"460e6e8f-ccc4-4952-934c-1d3229573074\" (UID: \"460e6e8f-ccc4-4952-934c-1d3229573074\") " Oct 11 11:05:30.679668 master-1 kubenswrapper[4771]: I1011 11:05:30.679626 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460e6e8f-ccc4-4952-934c-1d3229573074-kube-api-access-smwf6" (OuterVolumeSpecName: "kube-api-access-smwf6") pod "460e6e8f-ccc4-4952-934c-1d3229573074" (UID: "460e6e8f-ccc4-4952-934c-1d3229573074"). InnerVolumeSpecName "kube-api-access-smwf6". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:05:30.725789 master-1 kubenswrapper[4771]: I1011 11:05:30.725713 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-nb" (OuterVolumeSpecName: "ovsdbserver-nb") pod "460e6e8f-ccc4-4952-934c-1d3229573074" (UID: "460e6e8f-ccc4-4952-934c-1d3229573074"). InnerVolumeSpecName "ovsdbserver-nb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:05:30.733647 master-1 kubenswrapper[4771]: I1011 11:05:30.733534 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-networkers" (OuterVolumeSpecName: "networkers") pod "460e6e8f-ccc4-4952-934c-1d3229573074" (UID: "460e6e8f-ccc4-4952-934c-1d3229573074"). InnerVolumeSpecName "networkers". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:05:30.739047 master-1 kubenswrapper[4771]: I1011 11:05:30.738997 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-config" (OuterVolumeSpecName: "config") pod "460e6e8f-ccc4-4952-934c-1d3229573074" (UID: "460e6e8f-ccc4-4952-934c-1d3229573074"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:05:30.742422 master-1 kubenswrapper[4771]: I1011 11:05:30.742370 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-swift-storage-0" (OuterVolumeSpecName: "dns-swift-storage-0") pod "460e6e8f-ccc4-4952-934c-1d3229573074" (UID: "460e6e8f-ccc4-4952-934c-1d3229573074"). InnerVolumeSpecName "dns-swift-storage-0". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:05:30.749976 master-1 kubenswrapper[4771]: I1011 11:05:30.749936 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-sb" (OuterVolumeSpecName: "ovsdbserver-sb") pod "460e6e8f-ccc4-4952-934c-1d3229573074" (UID: "460e6e8f-ccc4-4952-934c-1d3229573074"). InnerVolumeSpecName "ovsdbserver-sb". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:05:30.753839 master-1 kubenswrapper[4771]: I1011 11:05:30.753763 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-svc" (OuterVolumeSpecName: "dns-svc") pod "460e6e8f-ccc4-4952-934c-1d3229573074" (UID: "460e6e8f-ccc4-4952-934c-1d3229573074"). InnerVolumeSpecName "dns-svc". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:05:30.777035 master-1 kubenswrapper[4771]: I1011 11:05:30.776985 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-smwf6\" (UniqueName: \"kubernetes.io/projected/460e6e8f-ccc4-4952-934c-1d3229573074-kube-api-access-smwf6\") on node \"master-1\" DevicePath \"\"" Oct 11 11:05:30.777035 master-1 kubenswrapper[4771]: I1011 11:05:30.777019 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-sb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-sb\") on node \"master-1\" DevicePath \"\"" Oct 11 11:05:30.777178 master-1 kubenswrapper[4771]: I1011 11:05:30.777048 4771 reconciler_common.go:293] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-config\") on node \"master-1\" DevicePath \"\"" Oct 11 11:05:30.777178 master-1 kubenswrapper[4771]: I1011 11:05:30.777060 4771 reconciler_common.go:293] "Volume detached for volume \"dns-svc\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-svc\") on node \"master-1\" DevicePath \"\"" Oct 11 11:05:30.777178 master-1 kubenswrapper[4771]: I1011 11:05:30.777069 4771 reconciler_common.go:293] "Volume detached for volume \"dns-swift-storage-0\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-dns-swift-storage-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:05:30.777178 master-1 kubenswrapper[4771]: I1011 11:05:30.777079 4771 reconciler_common.go:293] "Volume detached for volume \"networkers\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-networkers\") on node \"master-1\" DevicePath \"\"" Oct 11 11:05:30.777178 master-1 kubenswrapper[4771]: I1011 11:05:30.777087 4771 reconciler_common.go:293] "Volume detached for volume \"ovsdbserver-nb\" (UniqueName: \"kubernetes.io/configmap/460e6e8f-ccc4-4952-934c-1d3229573074-ovsdbserver-nb\") on node \"master-1\" DevicePath \"\"" Oct 11 11:05:30.856225 master-1 kubenswrapper[4771]: I1011 11:05:30.856030 4771 generic.go:334] "Generic (PLEG): container finished" podID="460e6e8f-ccc4-4952-934c-1d3229573074" containerID="47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47" exitCode=0 Oct 11 11:05:30.856225 master-1 kubenswrapper[4771]: I1011 11:05:30.856084 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" event={"ID":"460e6e8f-ccc4-4952-934c-1d3229573074","Type":"ContainerDied","Data":"47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47"} Oct 11 11:05:30.856225 master-1 kubenswrapper[4771]: I1011 11:05:30.856115 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" event={"ID":"460e6e8f-ccc4-4952-934c-1d3229573074","Type":"ContainerDied","Data":"e322bd0a72d963bfb151ddd53adda74fdeefdc268230c1597845f51e0682ee69"} Oct 11 11:05:30.856225 master-1 kubenswrapper[4771]: I1011 11:05:30.856134 4771 scope.go:117] "RemoveContainer" containerID="47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47" Oct 11 11:05:30.856786 master-1 kubenswrapper[4771]: I1011 11:05:30.856262 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/dnsmasq-dns-59dd57778c-jshg9" Oct 11 11:05:30.911173 master-1 kubenswrapper[4771]: I1011 11:05:30.910664 4771 scope.go:117] "RemoveContainer" containerID="4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f" Oct 11 11:05:30.933295 master-1 kubenswrapper[4771]: I1011 11:05:30.933225 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/dnsmasq-dns-59dd57778c-jshg9"] Oct 11 11:05:30.940017 master-1 kubenswrapper[4771]: I1011 11:05:30.939970 4771 scope.go:117] "RemoveContainer" containerID="47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47" Oct 11 11:05:30.940792 master-1 kubenswrapper[4771]: E1011 11:05:30.940728 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47\": container with ID starting with 47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47 not found: ID does not exist" containerID="47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47" Oct 11 11:05:30.940917 master-1 kubenswrapper[4771]: I1011 11:05:30.940793 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47"} err="failed to get container status \"47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47\": rpc error: code = NotFound desc = could not find container \"47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47\": container with ID starting with 47ed2540bea8eec9542f1728675554694e1d171301942dba021c31ee5d74ea47 not found: ID does not exist" Oct 11 11:05:30.940917 master-1 kubenswrapper[4771]: I1011 11:05:30.940811 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/dnsmasq-dns-59dd57778c-jshg9"] Oct 11 11:05:30.940917 master-1 kubenswrapper[4771]: I1011 11:05:30.940829 4771 scope.go:117] "RemoveContainer" containerID="4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f" Oct 11 11:05:30.941686 master-1 kubenswrapper[4771]: E1011 11:05:30.941626 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f\": container with ID starting with 4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f not found: ID does not exist" containerID="4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f" Oct 11 11:05:30.941793 master-1 kubenswrapper[4771]: I1011 11:05:30.941687 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f"} err="failed to get container status \"4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f\": rpc error: code = NotFound desc = could not find container \"4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f\": container with ID starting with 4cd14af125b21bd0750a27ba68476f9cdd511a9783b05c8056f3c380232ba59f not found: ID does not exist" Oct 11 11:05:30.993407 master-1 kubenswrapper[4771]: I1011 11:05:30.992141 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/dnsmasq-dns-f984c5fd9-vj7wp"] Oct 11 11:05:31.871927 master-1 kubenswrapper[4771]: I1011 11:05:31.871846 4771 generic.go:334] "Generic (PLEG): container finished" podID="73899937-c48a-4a79-9bc7-c5f4987908c3" containerID="20cfd66e6d311e07fd81153ee758ffd6072059205c2c6725635c8ceba6ca1b1e" exitCode=0 Oct 11 11:05:31.872723 master-1 kubenswrapper[4771]: I1011 11:05:31.871936 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" event={"ID":"73899937-c48a-4a79-9bc7-c5f4987908c3","Type":"ContainerDied","Data":"20cfd66e6d311e07fd81153ee758ffd6072059205c2c6725635c8ceba6ca1b1e"} Oct 11 11:05:31.872723 master-1 kubenswrapper[4771]: I1011 11:05:31.871997 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" event={"ID":"73899937-c48a-4a79-9bc7-c5f4987908c3","Type":"ContainerStarted","Data":"f10d01a7069e7e8cacab614b10bb215aa0cd2faf882c6288c6c10ab2628e80d2"} Oct 11 11:05:32.455481 master-1 kubenswrapper[4771]: I1011 11:05:32.455239 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="460e6e8f-ccc4-4952-934c-1d3229573074" path="/var/lib/kubelet/pods/460e6e8f-ccc4-4952-934c-1d3229573074/volumes" Oct 11 11:05:32.889820 master-1 kubenswrapper[4771]: I1011 11:05:32.889702 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" event={"ID":"73899937-c48a-4a79-9bc7-c5f4987908c3","Type":"ContainerStarted","Data":"df1802ddf887326037a580eaa64afb2b722cbb15026b43331527ade8e1ffb855"} Oct 11 11:05:32.890858 master-1 kubenswrapper[4771]: I1011 11:05:32.889935 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:32.927821 master-1 kubenswrapper[4771]: I1011 11:05:32.927725 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" podStartSLOduration=2.927702632 podStartE2EDuration="2.927702632s" podCreationTimestamp="2025-10-11 11:05:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 11:05:32.924530951 +0000 UTC m=+2364.898757432" watchObservedRunningTime="2025-10-11 11:05:32.927702632 +0000 UTC m=+2364.901929083" Oct 11 11:05:40.462652 master-1 kubenswrapper[4771]: I1011 11:05:40.462588 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openstack/dnsmasq-dns-f984c5fd9-vj7wp" Oct 11 11:05:51.059149 master-1 kubenswrapper[4771]: I1011 11:05:51.059035 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-api-a338-account-create-v4xzh"] Oct 11 11:05:51.072731 master-1 kubenswrapper[4771]: I1011 11:05:51.072636 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-api-a338-account-create-v4xzh"] Oct 11 11:05:52.447191 master-1 kubenswrapper[4771]: I1011 11:05:52.447120 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac9af7f-afc6-4d4d-9923-db14ac820459" path="/var/lib/kubelet/pods/7ac9af7f-afc6-4d4d-9923-db14ac820459/volumes" Oct 11 11:05:52.796446 master-1 kubenswrapper[4771]: I1011 11:05:52.796251 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/bootstrap-dataplane-edpm-hsdgd"] Oct 11 11:05:52.796815 master-1 kubenswrapper[4771]: E1011 11:05:52.796779 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="460e6e8f-ccc4-4952-934c-1d3229573074" containerName="init" Oct 11 11:05:52.796815 master-1 kubenswrapper[4771]: I1011 11:05:52.796806 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="460e6e8f-ccc4-4952-934c-1d3229573074" containerName="init" Oct 11 11:05:52.797033 master-1 kubenswrapper[4771]: E1011 11:05:52.796829 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="460e6e8f-ccc4-4952-934c-1d3229573074" containerName="dnsmasq-dns" Oct 11 11:05:52.797033 master-1 kubenswrapper[4771]: I1011 11:05:52.796844 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="460e6e8f-ccc4-4952-934c-1d3229573074" containerName="dnsmasq-dns" Oct 11 11:05:52.797194 master-1 kubenswrapper[4771]: I1011 11:05:52.797116 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="460e6e8f-ccc4-4952-934c-1d3229573074" containerName="dnsmasq-dns" Oct 11 11:05:52.798094 master-1 kubenswrapper[4771]: I1011 11:05:52.798063 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.801874 master-1 kubenswrapper[4771]: I1011 11:05:52.801808 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:05:52.802526 master-1 kubenswrapper[4771]: I1011 11:05:52.802469 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:05:52.802844 master-1 kubenswrapper[4771]: I1011 11:05:52.802480 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:05:52.812208 master-1 kubenswrapper[4771]: I1011 11:05:52.812147 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/bootstrap-dataplane-edpm-hsdgd"] Oct 11 11:05:52.845478 master-1 kubenswrapper[4771]: I1011 11:05:52.845400 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2frc\" (UniqueName: \"kubernetes.io/projected/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-kube-api-access-v2frc\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.845478 master-1 kubenswrapper[4771]: I1011 11:05:52.845473 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-inventory\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.845882 master-1 kubenswrapper[4771]: I1011 11:05:52.845567 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-bootstrap-combined-ca-bundle\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.845882 master-1 kubenswrapper[4771]: I1011 11:05:52.845606 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-ssh-key\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.948429 master-1 kubenswrapper[4771]: I1011 11:05:52.948369 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-v2frc\" (UniqueName: \"kubernetes.io/projected/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-kube-api-access-v2frc\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.948796 master-1 kubenswrapper[4771]: I1011 11:05:52.948442 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-inventory\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.948796 master-1 kubenswrapper[4771]: I1011 11:05:52.948514 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-bootstrap-combined-ca-bundle\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.948796 master-1 kubenswrapper[4771]: I1011 11:05:52.948567 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-ssh-key\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.952395 master-1 kubenswrapper[4771]: I1011 11:05:52.952330 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-inventory\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.958530 master-1 kubenswrapper[4771]: I1011 11:05:52.958423 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-bootstrap-combined-ca-bundle\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.958837 master-1 kubenswrapper[4771]: I1011 11:05:52.958642 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-ssh-key\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:52.978730 master-1 kubenswrapper[4771]: I1011 11:05:52.978650 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2frc\" (UniqueName: \"kubernetes.io/projected/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-kube-api-access-v2frc\") pod \"bootstrap-dataplane-edpm-hsdgd\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:53.156223 master-1 kubenswrapper[4771]: I1011 11:05:53.156129 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:05:53.786190 master-1 kubenswrapper[4771]: I1011 11:05:53.786099 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/bootstrap-dataplane-edpm-hsdgd"] Oct 11 11:05:53.787853 master-1 kubenswrapper[4771]: W1011 11:05:53.787799 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd3df1d_629a_4bbd_9ab4_7731e8928b01.slice/crio-a29718cbf38e7f25af3d27f643844ad043c8e1dc7c5a8dbda4e222f2e172deb6 WatchSource:0}: Error finding container a29718cbf38e7f25af3d27f643844ad043c8e1dc7c5a8dbda4e222f2e172deb6: Status 404 returned error can't find the container with id a29718cbf38e7f25af3d27f643844ad043c8e1dc7c5a8dbda4e222f2e172deb6 Oct 11 11:05:54.026014 master-1 kubenswrapper[4771]: I1011 11:05:54.025923 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/community-operators-j7glk"] Oct 11 11:05:54.031666 master-1 kubenswrapper[4771]: I1011 11:05:54.031565 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.044320 master-1 kubenswrapper[4771]: I1011 11:05:54.044222 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-j7glk"] Oct 11 11:05:54.147502 master-1 kubenswrapper[4771]: I1011 11:05:54.147425 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/bootstrap-dataplane-edpm-hsdgd" event={"ID":"7bd3df1d-629a-4bbd-9ab4-7731e8928b01","Type":"ContainerStarted","Data":"a29718cbf38e7f25af3d27f643844ad043c8e1dc7c5a8dbda4e222f2e172deb6"} Oct 11 11:05:54.177979 master-1 kubenswrapper[4771]: I1011 11:05:54.177929 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8pb\" (UniqueName: \"kubernetes.io/projected/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-kube-api-access-nc8pb\") pod \"community-operators-j7glk\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.178129 master-1 kubenswrapper[4771]: I1011 11:05:54.177994 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-utilities\") pod \"community-operators-j7glk\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.178129 master-1 kubenswrapper[4771]: I1011 11:05:54.178063 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-catalog-content\") pod \"community-operators-j7glk\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.280309 master-1 kubenswrapper[4771]: I1011 11:05:54.280232 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-catalog-content\") pod \"community-operators-j7glk\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.280705 master-1 kubenswrapper[4771]: I1011 11:05:54.280470 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8pb\" (UniqueName: \"kubernetes.io/projected/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-kube-api-access-nc8pb\") pod \"community-operators-j7glk\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.280705 master-1 kubenswrapper[4771]: I1011 11:05:54.280538 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-utilities\") pod \"community-operators-j7glk\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.281533 master-1 kubenswrapper[4771]: I1011 11:05:54.281390 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-utilities\") pod \"community-operators-j7glk\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.281801 master-1 kubenswrapper[4771]: I1011 11:05:54.281613 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-catalog-content\") pod \"community-operators-j7glk\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.313347 master-1 kubenswrapper[4771]: I1011 11:05:54.313262 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8pb\" (UniqueName: \"kubernetes.io/projected/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-kube-api-access-nc8pb\") pod \"community-operators-j7glk\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.368961 master-1 kubenswrapper[4771]: I1011 11:05:54.368402 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:05:54.885758 master-1 kubenswrapper[4771]: I1011 11:05:54.884930 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-j7glk"] Oct 11 11:05:55.158661 master-1 kubenswrapper[4771]: I1011 11:05:55.158612 4771 generic.go:334] "Generic (PLEG): container finished" podID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerID="28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699" exitCode=0 Oct 11 11:05:55.158784 master-1 kubenswrapper[4771]: I1011 11:05:55.158674 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-j7glk" event={"ID":"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2","Type":"ContainerDied","Data":"28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699"} Oct 11 11:05:55.158784 master-1 kubenswrapper[4771]: I1011 11:05:55.158709 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-j7glk" event={"ID":"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2","Type":"ContainerStarted","Data":"39291eb0bb4dddc8b1a55028f1e708f4628e007e9d0a4908a7a947f1a79a7a0f"} Oct 11 11:05:56.175965 master-1 kubenswrapper[4771]: I1011 11:05:56.175894 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-j7glk" event={"ID":"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2","Type":"ContainerStarted","Data":"45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7"} Oct 11 11:05:57.188854 master-1 kubenswrapper[4771]: I1011 11:05:57.188749 4771 generic.go:334] "Generic (PLEG): container finished" podID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerID="45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7" exitCode=0 Oct 11 11:05:57.188854 master-1 kubenswrapper[4771]: I1011 11:05:57.188847 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-j7glk" event={"ID":"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2","Type":"ContainerDied","Data":"45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7"} Oct 11 11:06:04.260074 master-1 kubenswrapper[4771]: I1011 11:06:04.259877 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-j7glk" event={"ID":"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2","Type":"ContainerStarted","Data":"a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d"} Oct 11 11:06:04.262555 master-1 kubenswrapper[4771]: I1011 11:06:04.262437 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/bootstrap-dataplane-edpm-hsdgd" event={"ID":"7bd3df1d-629a-4bbd-9ab4-7731e8928b01","Type":"ContainerStarted","Data":"b6fe2bafc6abee3dc8fb39a96bf66118ada768aa6b65ffb7178ecf81e4862ef0"} Oct 11 11:06:04.316072 master-1 kubenswrapper[4771]: I1011 11:06:04.315944 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/community-operators-j7glk" podStartSLOduration=2.769382736 podStartE2EDuration="11.315921513s" podCreationTimestamp="2025-10-11 11:05:53 +0000 UTC" firstStartedPulling="2025-10-11 11:05:55.160746598 +0000 UTC m=+2387.134973039" lastFinishedPulling="2025-10-11 11:06:03.707285375 +0000 UTC m=+2395.681511816" observedRunningTime="2025-10-11 11:06:04.307960806 +0000 UTC m=+2396.282187277" watchObservedRunningTime="2025-10-11 11:06:04.315921513 +0000 UTC m=+2396.290147964" Oct 11 11:06:04.353320 master-1 kubenswrapper[4771]: I1011 11:06:04.353205 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/bootstrap-dataplane-edpm-hsdgd" podStartSLOduration=2.412388001 podStartE2EDuration="12.353182739s" podCreationTimestamp="2025-10-11 11:05:52 +0000 UTC" firstStartedPulling="2025-10-11 11:05:53.792009667 +0000 UTC m=+2385.766236108" lastFinishedPulling="2025-10-11 11:06:03.732804405 +0000 UTC m=+2395.707030846" observedRunningTime="2025-10-11 11:06:04.347511577 +0000 UTC m=+2396.321738048" watchObservedRunningTime="2025-10-11 11:06:04.353182739 +0000 UTC m=+2396.327409190" Oct 11 11:06:04.369533 master-1 kubenswrapper[4771]: I1011 11:06:04.369462 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:06:04.369671 master-1 kubenswrapper[4771]: I1011 11:06:04.369547 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:06:05.416761 master-1 kubenswrapper[4771]: I1011 11:06:05.416670 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-marketplace/community-operators-j7glk" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerName="registry-server" probeResult="failure" output=< Oct 11 11:06:05.416761 master-1 kubenswrapper[4771]: timeout: failed to connect service ":50051" within 1s Oct 11 11:06:05.416761 master-1 kubenswrapper[4771]: > Oct 11 11:06:11.075798 master-1 kubenswrapper[4771]: I1011 11:06:11.075616 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/aodh-db-create-sz8dm"] Oct 11 11:06:11.092300 master-1 kubenswrapper[4771]: I1011 11:06:11.092194 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/aodh-db-create-sz8dm"] Oct 11 11:06:12.455799 master-1 kubenswrapper[4771]: I1011 11:06:12.455701 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa024267-404c-497a-a798-3a371608b678" path="/var/lib/kubelet/pods/fa024267-404c-497a-a798-3a371608b678/volumes" Oct 11 11:06:14.447038 master-1 kubenswrapper[4771]: I1011 11:06:14.446972 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:06:14.516848 master-1 kubenswrapper[4771]: I1011 11:06:14.516772 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:06:14.711937 master-1 kubenswrapper[4771]: I1011 11:06:14.711794 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-j7glk"] Oct 11 11:06:16.417062 master-1 kubenswrapper[4771]: I1011 11:06:16.416943 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/community-operators-j7glk" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerName="registry-server" containerID="cri-o://a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d" gracePeriod=2 Oct 11 11:06:17.031147 master-1 kubenswrapper[4771]: I1011 11:06:17.031068 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:06:17.080504 master-1 kubenswrapper[4771]: I1011 11:06:17.076891 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc8pb\" (UniqueName: \"kubernetes.io/projected/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-kube-api-access-nc8pb\") pod \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " Oct 11 11:06:17.080504 master-1 kubenswrapper[4771]: I1011 11:06:17.077114 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-utilities\") pod \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " Oct 11 11:06:17.080504 master-1 kubenswrapper[4771]: I1011 11:06:17.077179 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-catalog-content\") pod \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\" (UID: \"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2\") " Oct 11 11:06:17.080504 master-1 kubenswrapper[4771]: I1011 11:06:17.078939 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-utilities" (OuterVolumeSpecName: "utilities") pod "f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" (UID: "f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:06:17.082946 master-1 kubenswrapper[4771]: I1011 11:06:17.082837 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-kube-api-access-nc8pb" (OuterVolumeSpecName: "kube-api-access-nc8pb") pod "f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" (UID: "f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2"). InnerVolumeSpecName "kube-api-access-nc8pb". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:06:17.139768 master-1 kubenswrapper[4771]: I1011 11:06:17.139671 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" (UID: "f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:06:17.182522 master-1 kubenswrapper[4771]: I1011 11:06:17.182440 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-nc8pb\" (UniqueName: \"kubernetes.io/projected/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-kube-api-access-nc8pb\") on node \"master-1\" DevicePath \"\"" Oct 11 11:06:17.182522 master-1 kubenswrapper[4771]: I1011 11:06:17.182514 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:06:17.182824 master-1 kubenswrapper[4771]: I1011 11:06:17.182536 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:06:17.431648 master-1 kubenswrapper[4771]: I1011 11:06:17.431576 4771 generic.go:334] "Generic (PLEG): container finished" podID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerID="a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d" exitCode=0 Oct 11 11:06:17.431648 master-1 kubenswrapper[4771]: I1011 11:06:17.431647 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-j7glk" event={"ID":"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2","Type":"ContainerDied","Data":"a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d"} Oct 11 11:06:17.432304 master-1 kubenswrapper[4771]: I1011 11:06:17.431684 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-j7glk" Oct 11 11:06:17.432304 master-1 kubenswrapper[4771]: I1011 11:06:17.431729 4771 scope.go:117] "RemoveContainer" containerID="a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d" Oct 11 11:06:17.432304 master-1 kubenswrapper[4771]: I1011 11:06:17.431712 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-j7glk" event={"ID":"f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2","Type":"ContainerDied","Data":"39291eb0bb4dddc8b1a55028f1e708f4628e007e9d0a4908a7a947f1a79a7a0f"} Oct 11 11:06:17.457043 master-1 kubenswrapper[4771]: I1011 11:06:17.454724 4771 scope.go:117] "RemoveContainer" containerID="45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7" Oct 11 11:06:17.486713 master-1 kubenswrapper[4771]: I1011 11:06:17.486653 4771 scope.go:117] "RemoveContainer" containerID="28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699" Oct 11 11:06:17.503179 master-1 kubenswrapper[4771]: I1011 11:06:17.503113 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-j7glk"] Oct 11 11:06:17.516295 master-1 kubenswrapper[4771]: I1011 11:06:17.516229 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/community-operators-j7glk"] Oct 11 11:06:17.518880 master-1 kubenswrapper[4771]: I1011 11:06:17.518834 4771 scope.go:117] "RemoveContainer" containerID="a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d" Oct 11 11:06:17.519607 master-1 kubenswrapper[4771]: E1011 11:06:17.519555 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d\": container with ID starting with a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d not found: ID does not exist" containerID="a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d" Oct 11 11:06:17.519675 master-1 kubenswrapper[4771]: I1011 11:06:17.519619 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d"} err="failed to get container status \"a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d\": rpc error: code = NotFound desc = could not find container \"a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d\": container with ID starting with a7251ba4d7d22653d6ccb9ee7c097e5d20e42d6656045bba5df4b68d8f18c09d not found: ID does not exist" Oct 11 11:06:17.519675 master-1 kubenswrapper[4771]: I1011 11:06:17.519649 4771 scope.go:117] "RemoveContainer" containerID="45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7" Oct 11 11:06:17.520175 master-1 kubenswrapper[4771]: E1011 11:06:17.520118 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7\": container with ID starting with 45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7 not found: ID does not exist" containerID="45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7" Oct 11 11:06:17.520260 master-1 kubenswrapper[4771]: I1011 11:06:17.520182 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7"} err="failed to get container status \"45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7\": rpc error: code = NotFound desc = could not find container \"45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7\": container with ID starting with 45ae6a7716efdd3aea461d5292cdae5828f0ecbb445a641e81b3a2c67ebda8e7 not found: ID does not exist" Oct 11 11:06:17.520260 master-1 kubenswrapper[4771]: I1011 11:06:17.520215 4771 scope.go:117] "RemoveContainer" containerID="28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699" Oct 11 11:06:17.520703 master-1 kubenswrapper[4771]: E1011 11:06:17.520645 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699\": container with ID starting with 28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699 not found: ID does not exist" containerID="28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699" Oct 11 11:06:17.520807 master-1 kubenswrapper[4771]: I1011 11:06:17.520689 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699"} err="failed to get container status \"28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699\": rpc error: code = NotFound desc = could not find container \"28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699\": container with ID starting with 28f51367dd6d11abe17cd09270952a1c6d945d37acbc8713d9275ca61fa60699 not found: ID does not exist" Oct 11 11:06:18.449903 master-1 kubenswrapper[4771]: I1011 11:06:18.449800 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" path="/var/lib/kubelet/pods/f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2/volumes" Oct 11 11:06:20.062891 master-1 kubenswrapper[4771]: I1011 11:06:20.061909 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-cell0-conductor-db-sync-chcrd"] Oct 11 11:06:20.072004 master-1 kubenswrapper[4771]: I1011 11:06:20.071939 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-cell0-conductor-db-sync-chcrd"] Oct 11 11:06:20.448026 master-1 kubenswrapper[4771]: I1011 11:06:20.447912 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38267a66-0ebd-44ab-bc7f-cd5703503b74" path="/var/lib/kubelet/pods/38267a66-0ebd-44ab-bc7f-cd5703503b74/volumes" Oct 11 11:06:23.949212 master-1 kubenswrapper[4771]: I1011 11:06:23.949123 4771 scope.go:117] "RemoveContainer" containerID="558ef049f7336b936f032c4d3e3115131e36703eb572e93323b57a5fd484ff9e" Oct 11 11:06:23.988856 master-1 kubenswrapper[4771]: I1011 11:06:23.988509 4771 scope.go:117] "RemoveContainer" containerID="e1ee0992af169f3773493c300780fafe6521ac72bd4a220402d3338c4c92c6fb" Oct 11 11:06:24.027167 master-1 kubenswrapper[4771]: I1011 11:06:24.026879 4771 scope.go:117] "RemoveContainer" containerID="5d17ff04cecf6e6d74e4dc9eda892c61efec1bfa4b25f713f94a437e54e6aeed" Oct 11 11:06:24.062035 master-1 kubenswrapper[4771]: I1011 11:06:24.061939 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/aodh-1a24-account-create-pb6gd"] Oct 11 11:06:24.073812 master-1 kubenswrapper[4771]: I1011 11:06:24.073735 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/aodh-1a24-account-create-pb6gd"] Oct 11 11:06:24.456215 master-1 kubenswrapper[4771]: I1011 11:06:24.455978 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b670525b-9ca9-419c-858b-6bb2a2303cf6" path="/var/lib/kubelet/pods/b670525b-9ca9-419c-858b-6bb2a2303cf6/volumes" Oct 11 11:06:43.079324 master-1 kubenswrapper[4771]: I1011 11:06:43.079189 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/aodh-db-sync-tn8xz"] Oct 11 11:06:43.096025 master-1 kubenswrapper[4771]: I1011 11:06:43.095931 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/aodh-db-sync-tn8xz"] Oct 11 11:06:44.050680 master-1 kubenswrapper[4771]: I1011 11:06:44.050614 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-cell0-cell-mapping-2kt7k"] Oct 11 11:06:44.057775 master-1 kubenswrapper[4771]: I1011 11:06:44.057733 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-cell0-cell-mapping-2kt7k"] Oct 11 11:06:44.457099 master-1 kubenswrapper[4771]: I1011 11:06:44.457029 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de492fb-5249-49e2-a327-756234aa92bd" path="/var/lib/kubelet/pods/3de492fb-5249-49e2-a327-756234aa92bd/volumes" Oct 11 11:06:44.458049 master-1 kubenswrapper[4771]: I1011 11:06:44.458020 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f85d5cfa-8073-4bbf-9eff-78fde719dadf" path="/var/lib/kubelet/pods/f85d5cfa-8073-4bbf-9eff-78fde719dadf/volumes" Oct 11 11:06:46.051268 master-1 kubenswrapper[4771]: I1011 11:06:46.051200 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-cell1-conductor-db-sync-tgg64"] Oct 11 11:06:46.066025 master-1 kubenswrapper[4771]: I1011 11:06:46.065923 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-cell1-conductor-db-sync-tgg64"] Oct 11 11:06:46.456385 master-1 kubenswrapper[4771]: I1011 11:06:46.456260 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5e37e3-9afd-4ff3-b992-1e6c28a986ad" path="/var/lib/kubelet/pods/5b5e37e3-9afd-4ff3-b992-1e6c28a986ad/volumes" Oct 11 11:06:58.997608 master-1 kubenswrapper[4771]: I1011 11:06:58.997515 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-operators-ml7zj"] Oct 11 11:06:58.998659 master-1 kubenswrapper[4771]: E1011 11:06:58.998034 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerName="extract-content" Oct 11 11:06:58.998659 master-1 kubenswrapper[4771]: I1011 11:06:58.998060 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerName="extract-content" Oct 11 11:06:58.998659 master-1 kubenswrapper[4771]: E1011 11:06:58.998086 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerName="registry-server" Oct 11 11:06:58.998659 master-1 kubenswrapper[4771]: I1011 11:06:58.998099 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerName="registry-server" Oct 11 11:06:58.998659 master-1 kubenswrapper[4771]: E1011 11:06:58.998152 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerName="extract-utilities" Oct 11 11:06:58.998659 master-1 kubenswrapper[4771]: I1011 11:06:58.998166 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerName="extract-utilities" Oct 11 11:06:58.998659 master-1 kubenswrapper[4771]: I1011 11:06:58.998470 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="f8b4e7d7-c57b-4b03-b229-91c6ceb0cda2" containerName="registry-server" Oct 11 11:06:59.000922 master-1 kubenswrapper[4771]: I1011 11:06:59.000863 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.021765 master-1 kubenswrapper[4771]: I1011 11:06:59.021680 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-ml7zj"] Oct 11 11:06:59.159806 master-1 kubenswrapper[4771]: I1011 11:06:59.159685 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-utilities\") pod \"redhat-operators-ml7zj\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.160056 master-1 kubenswrapper[4771]: I1011 11:06:59.159923 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xks\" (UniqueName: \"kubernetes.io/projected/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-kube-api-access-b9xks\") pod \"redhat-operators-ml7zj\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.160117 master-1 kubenswrapper[4771]: I1011 11:06:59.160095 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-catalog-content\") pod \"redhat-operators-ml7zj\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.275616 master-1 kubenswrapper[4771]: I1011 11:06:59.274441 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-utilities\") pod \"redhat-operators-ml7zj\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.275616 master-1 kubenswrapper[4771]: I1011 11:06:59.274746 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xks\" (UniqueName: \"kubernetes.io/projected/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-kube-api-access-b9xks\") pod \"redhat-operators-ml7zj\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.275616 master-1 kubenswrapper[4771]: I1011 11:06:59.275026 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-utilities\") pod \"redhat-operators-ml7zj\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.275616 master-1 kubenswrapper[4771]: I1011 11:06:59.275072 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-catalog-content\") pod \"redhat-operators-ml7zj\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.275616 master-1 kubenswrapper[4771]: I1011 11:06:59.275373 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-catalog-content\") pod \"redhat-operators-ml7zj\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.298291 master-1 kubenswrapper[4771]: I1011 11:06:59.298238 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xks\" (UniqueName: \"kubernetes.io/projected/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-kube-api-access-b9xks\") pod \"redhat-operators-ml7zj\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.336164 master-1 kubenswrapper[4771]: I1011 11:06:59.336101 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:06:59.911410 master-1 kubenswrapper[4771]: I1011 11:06:59.911338 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-ml7zj"] Oct 11 11:06:59.925108 master-1 kubenswrapper[4771]: I1011 11:06:59.925055 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-ml7zj" event={"ID":"975d8dbf-78fe-4498-ba3c-c77b71e3d13c","Type":"ContainerStarted","Data":"ef3300826966f8ec98a66b7079eaad081f80ea7c8396e36a817549fc64d17745"} Oct 11 11:07:00.936022 master-1 kubenswrapper[4771]: I1011 11:07:00.935937 4771 generic.go:334] "Generic (PLEG): container finished" podID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerID="7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820" exitCode=0 Oct 11 11:07:00.936022 master-1 kubenswrapper[4771]: I1011 11:07:00.936017 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-ml7zj" event={"ID":"975d8dbf-78fe-4498-ba3c-c77b71e3d13c","Type":"ContainerDied","Data":"7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820"} Oct 11 11:07:02.967163 master-1 kubenswrapper[4771]: I1011 11:07:02.967048 4771 generic.go:334] "Generic (PLEG): container finished" podID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerID="5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d" exitCode=0 Oct 11 11:07:02.967163 master-1 kubenswrapper[4771]: I1011 11:07:02.967149 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-ml7zj" event={"ID":"975d8dbf-78fe-4498-ba3c-c77b71e3d13c","Type":"ContainerDied","Data":"5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d"} Oct 11 11:07:03.981561 master-1 kubenswrapper[4771]: I1011 11:07:03.981389 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-ml7zj" event={"ID":"975d8dbf-78fe-4498-ba3c-c77b71e3d13c","Type":"ContainerStarted","Data":"0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947"} Oct 11 11:07:04.046476 master-1 kubenswrapper[4771]: I1011 11:07:04.046328 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-operators-ml7zj" podStartSLOduration=3.616338558 podStartE2EDuration="6.046298126s" podCreationTimestamp="2025-10-11 11:06:58 +0000 UTC" firstStartedPulling="2025-10-11 11:07:00.938800639 +0000 UTC m=+2452.913027100" lastFinishedPulling="2025-10-11 11:07:03.368760227 +0000 UTC m=+2455.342986668" observedRunningTime="2025-10-11 11:07:04.036207657 +0000 UTC m=+2456.010434188" watchObservedRunningTime="2025-10-11 11:07:04.046298126 +0000 UTC m=+2456.020524607" Oct 11 11:07:09.337088 master-1 kubenswrapper[4771]: I1011 11:07:09.337017 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:07:09.338505 master-1 kubenswrapper[4771]: I1011 11:07:09.337410 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:07:10.406552 master-1 kubenswrapper[4771]: I1011 11:07:10.406452 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-marketplace/redhat-operators-ml7zj" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerName="registry-server" probeResult="failure" output=< Oct 11 11:07:10.406552 master-1 kubenswrapper[4771]: timeout: failed to connect service ":50051" within 1s Oct 11 11:07:10.406552 master-1 kubenswrapper[4771]: > Oct 11 11:07:19.422745 master-1 kubenswrapper[4771]: I1011 11:07:19.422634 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:07:19.509394 master-1 kubenswrapper[4771]: I1011 11:07:19.509274 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:07:19.696575 master-1 kubenswrapper[4771]: I1011 11:07:19.696332 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-ml7zj"] Oct 11 11:07:21.190478 master-1 kubenswrapper[4771]: I1011 11:07:21.190295 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-operators-ml7zj" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerName="registry-server" containerID="cri-o://0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947" gracePeriod=2 Oct 11 11:07:21.842681 master-1 kubenswrapper[4771]: I1011 11:07:21.842625 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:07:22.039638 master-1 kubenswrapper[4771]: I1011 11:07:22.039476 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-utilities\") pod \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " Oct 11 11:07:22.039638 master-1 kubenswrapper[4771]: I1011 11:07:22.039579 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9xks\" (UniqueName: \"kubernetes.io/projected/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-kube-api-access-b9xks\") pod \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " Oct 11 11:07:22.039973 master-1 kubenswrapper[4771]: I1011 11:07:22.039659 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-catalog-content\") pod \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\" (UID: \"975d8dbf-78fe-4498-ba3c-c77b71e3d13c\") " Oct 11 11:07:22.041068 master-1 kubenswrapper[4771]: I1011 11:07:22.040997 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-utilities" (OuterVolumeSpecName: "utilities") pod "975d8dbf-78fe-4498-ba3c-c77b71e3d13c" (UID: "975d8dbf-78fe-4498-ba3c-c77b71e3d13c"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:07:22.056892 master-1 kubenswrapper[4771]: I1011 11:07:22.056839 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-kube-api-access-b9xks" (OuterVolumeSpecName: "kube-api-access-b9xks") pod "975d8dbf-78fe-4498-ba3c-c77b71e3d13c" (UID: "975d8dbf-78fe-4498-ba3c-c77b71e3d13c"). InnerVolumeSpecName "kube-api-access-b9xks". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:07:22.142063 master-1 kubenswrapper[4771]: I1011 11:07:22.142008 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:07:22.142063 master-1 kubenswrapper[4771]: I1011 11:07:22.142051 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-b9xks\" (UniqueName: \"kubernetes.io/projected/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-kube-api-access-b9xks\") on node \"master-1\" DevicePath \"\"" Oct 11 11:07:22.176264 master-1 kubenswrapper[4771]: I1011 11:07:22.176199 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "975d8dbf-78fe-4498-ba3c-c77b71e3d13c" (UID: "975d8dbf-78fe-4498-ba3c-c77b71e3d13c"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:07:22.204146 master-1 kubenswrapper[4771]: I1011 11:07:22.204082 4771 generic.go:334] "Generic (PLEG): container finished" podID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerID="0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947" exitCode=0 Oct 11 11:07:22.204146 master-1 kubenswrapper[4771]: I1011 11:07:22.204136 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-ml7zj" event={"ID":"975d8dbf-78fe-4498-ba3c-c77b71e3d13c","Type":"ContainerDied","Data":"0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947"} Oct 11 11:07:22.204619 master-1 kubenswrapper[4771]: I1011 11:07:22.204178 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-ml7zj" event={"ID":"975d8dbf-78fe-4498-ba3c-c77b71e3d13c","Type":"ContainerDied","Data":"ef3300826966f8ec98a66b7079eaad081f80ea7c8396e36a817549fc64d17745"} Oct 11 11:07:22.204619 master-1 kubenswrapper[4771]: I1011 11:07:22.204202 4771 scope.go:117] "RemoveContainer" containerID="0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947" Oct 11 11:07:22.204619 master-1 kubenswrapper[4771]: I1011 11:07:22.204213 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-ml7zj" Oct 11 11:07:22.228625 master-1 kubenswrapper[4771]: I1011 11:07:22.228572 4771 scope.go:117] "RemoveContainer" containerID="5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d" Oct 11 11:07:22.245555 master-1 kubenswrapper[4771]: I1011 11:07:22.244908 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/975d8dbf-78fe-4498-ba3c-c77b71e3d13c-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:07:22.253603 master-1 kubenswrapper[4771]: I1011 11:07:22.253419 4771 scope.go:117] "RemoveContainer" containerID="7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820" Oct 11 11:07:22.260200 master-1 kubenswrapper[4771]: I1011 11:07:22.260126 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-ml7zj"] Oct 11 11:07:22.269953 master-1 kubenswrapper[4771]: I1011 11:07:22.268500 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-operators-ml7zj"] Oct 11 11:07:22.319345 master-1 kubenswrapper[4771]: I1011 11:07:22.319275 4771 scope.go:117] "RemoveContainer" containerID="0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947" Oct 11 11:07:22.320145 master-1 kubenswrapper[4771]: E1011 11:07:22.319992 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947\": container with ID starting with 0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947 not found: ID does not exist" containerID="0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947" Oct 11 11:07:22.320145 master-1 kubenswrapper[4771]: I1011 11:07:22.320051 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947"} err="failed to get container status \"0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947\": rpc error: code = NotFound desc = could not find container \"0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947\": container with ID starting with 0fabae86416d42993c56f8001eca8e0c83d909a12fa2532f9d5f6f9861d1e947 not found: ID does not exist" Oct 11 11:07:22.320145 master-1 kubenswrapper[4771]: I1011 11:07:22.320087 4771 scope.go:117] "RemoveContainer" containerID="5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d" Oct 11 11:07:22.320687 master-1 kubenswrapper[4771]: E1011 11:07:22.320639 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d\": container with ID starting with 5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d not found: ID does not exist" containerID="5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d" Oct 11 11:07:22.320744 master-1 kubenswrapper[4771]: I1011 11:07:22.320694 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d"} err="failed to get container status \"5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d\": rpc error: code = NotFound desc = could not find container \"5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d\": container with ID starting with 5729cff9675c3fe08ce7639f6deb94291b153462aad880a76ad3b64a18541b6d not found: ID does not exist" Oct 11 11:07:22.320744 master-1 kubenswrapper[4771]: I1011 11:07:22.320730 4771 scope.go:117] "RemoveContainer" containerID="7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820" Oct 11 11:07:22.321260 master-1 kubenswrapper[4771]: E1011 11:07:22.321194 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820\": container with ID starting with 7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820 not found: ID does not exist" containerID="7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820" Oct 11 11:07:22.321314 master-1 kubenswrapper[4771]: I1011 11:07:22.321273 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820"} err="failed to get container status \"7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820\": rpc error: code = NotFound desc = could not find container \"7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820\": container with ID starting with 7f9d75f8079234d810d11902757682947dc5fb16ad7718bd4f50ddbf03868820 not found: ID does not exist" Oct 11 11:07:22.449318 master-1 kubenswrapper[4771]: I1011 11:07:22.449205 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" path="/var/lib/kubelet/pods/975d8dbf-78fe-4498-ba3c-c77b71e3d13c/volumes" Oct 11 11:07:24.218382 master-1 kubenswrapper[4771]: I1011 11:07:24.218269 4771 scope.go:117] "RemoveContainer" containerID="34c012fefebf03c137c3d264726e9a32c974159496d4bf0d0a4dad6dcdf4c655" Oct 11 11:07:24.253730 master-1 kubenswrapper[4771]: I1011 11:07:24.252795 4771 scope.go:117] "RemoveContainer" containerID="03e80f5bdb6844a3112427ed3612b145765c86f689c582771359401e14c9758e" Oct 11 11:07:24.355016 master-1 kubenswrapper[4771]: I1011 11:07:24.354940 4771 scope.go:117] "RemoveContainer" containerID="38fe7e740cc7430b1900679565564cc35f6e1964bf7c4a238c960c0377445331" Oct 11 11:07:24.388737 master-1 kubenswrapper[4771]: I1011 11:07:24.388664 4771 scope.go:117] "RemoveContainer" containerID="af56a7e4623de207ef8289e7bba0d65eef5da9d57f459e288f321109c3a8e4f3" Oct 11 11:07:25.074069 master-1 kubenswrapper[4771]: I1011 11:07:25.073974 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-cell1-host-discover-5f556"] Oct 11 11:07:25.080020 master-1 kubenswrapper[4771]: I1011 11:07:25.079930 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-cell1-host-discover-5f556"] Oct 11 11:07:26.452550 master-1 kubenswrapper[4771]: I1011 11:07:26.452474 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae49cc63-d351-440f-9334-4ef2550565a2" path="/var/lib/kubelet/pods/ae49cc63-d351-440f-9334-4ef2550565a2/volumes" Oct 11 11:07:27.063532 master-1 kubenswrapper[4771]: I1011 11:07:27.063439 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/nova-cell1-cell-mapping-bwgtz"] Oct 11 11:07:27.073745 master-1 kubenswrapper[4771]: I1011 11:07:27.073656 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/nova-cell1-cell-mapping-bwgtz"] Oct 11 11:07:28.451640 master-1 kubenswrapper[4771]: I1011 11:07:28.451583 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="709c362a-6ace-46bf-9f94-86852f78f6f2" path="/var/lib/kubelet/pods/709c362a-6ace-46bf-9f94-86852f78f6f2/volumes" Oct 11 11:08:24.595133 master-1 kubenswrapper[4771]: I1011 11:08:24.595055 4771 scope.go:117] "RemoveContainer" containerID="99d58d9d6b8b62fa18ae8ba7508466dad2a9761e505b9274423ecba095a9de64" Oct 11 11:08:24.658949 master-1 kubenswrapper[4771]: I1011 11:08:24.658890 4771 scope.go:117] "RemoveContainer" containerID="48e35ef26a01bac7444e96fa2a9fa3fe07bd9eb6b20913ec8c1c945288cc11bc" Oct 11 11:10:06.962971 master-1 kubenswrapper[4771]: I1011 11:10:06.962053 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/configure-network-networker-deploy-networkers-vkj22"] Oct 11 11:10:06.965369 master-1 kubenswrapper[4771]: E1011 11:10:06.965318 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerName="extract-content" Oct 11 11:10:06.965443 master-1 kubenswrapper[4771]: I1011 11:10:06.965412 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerName="extract-content" Oct 11 11:10:06.965481 master-1 kubenswrapper[4771]: E1011 11:10:06.965469 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerName="registry-server" Oct 11 11:10:06.965481 master-1 kubenswrapper[4771]: I1011 11:10:06.965480 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerName="registry-server" Oct 11 11:10:06.965541 master-1 kubenswrapper[4771]: E1011 11:10:06.965498 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerName="extract-utilities" Oct 11 11:10:06.965541 master-1 kubenswrapper[4771]: I1011 11:10:06.965508 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerName="extract-utilities" Oct 11 11:10:06.965772 master-1 kubenswrapper[4771]: I1011 11:10:06.965739 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="975d8dbf-78fe-4498-ba3c-c77b71e3d13c" containerName="registry-server" Oct 11 11:10:06.967021 master-1 kubenswrapper[4771]: I1011 11:10:06.966976 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:06.970048 master-1 kubenswrapper[4771]: I1011 11:10:06.969985 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-networkers" Oct 11 11:10:06.986044 master-1 kubenswrapper[4771]: I1011 11:10:06.985946 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/configure-network-networker-deploy-networkers-vkj22"] Oct 11 11:10:07.031662 master-1 kubenswrapper[4771]: I1011 11:10:07.031565 4771 generic.go:334] "Generic (PLEG): container finished" podID="7bd3df1d-629a-4bbd-9ab4-7731e8928b01" containerID="b6fe2bafc6abee3dc8fb39a96bf66118ada768aa6b65ffb7178ecf81e4862ef0" exitCode=0 Oct 11 11:10:07.031662 master-1 kubenswrapper[4771]: I1011 11:10:07.031650 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/bootstrap-dataplane-edpm-hsdgd" event={"ID":"7bd3df1d-629a-4bbd-9ab4-7731e8928b01","Type":"ContainerDied","Data":"b6fe2bafc6abee3dc8fb39a96bf66118ada768aa6b65ffb7178ecf81e4862ef0"} Oct 11 11:10:07.093268 master-1 kubenswrapper[4771]: I1011 11:10:07.093155 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-ssh-key\") pod \"configure-network-networker-deploy-networkers-vkj22\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:07.093602 master-1 kubenswrapper[4771]: I1011 11:10:07.093321 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-inventory\") pod \"configure-network-networker-deploy-networkers-vkj22\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:07.093602 master-1 kubenswrapper[4771]: I1011 11:10:07.093428 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc4dv\" (UniqueName: \"kubernetes.io/projected/23431f0d-ef6f-4620-a467-15eda9b19df4-kube-api-access-sc4dv\") pod \"configure-network-networker-deploy-networkers-vkj22\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:07.195446 master-1 kubenswrapper[4771]: I1011 11:10:07.195351 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-sc4dv\" (UniqueName: \"kubernetes.io/projected/23431f0d-ef6f-4620-a467-15eda9b19df4-kube-api-access-sc4dv\") pod \"configure-network-networker-deploy-networkers-vkj22\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:07.195677 master-1 kubenswrapper[4771]: I1011 11:10:07.195542 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-ssh-key\") pod \"configure-network-networker-deploy-networkers-vkj22\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:07.195677 master-1 kubenswrapper[4771]: I1011 11:10:07.195641 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-inventory\") pod \"configure-network-networker-deploy-networkers-vkj22\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:07.225136 master-1 kubenswrapper[4771]: I1011 11:10:07.224990 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-ssh-key\") pod \"configure-network-networker-deploy-networkers-vkj22\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:07.228775 master-1 kubenswrapper[4771]: I1011 11:10:07.228719 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-inventory\") pod \"configure-network-networker-deploy-networkers-vkj22\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:07.231242 master-1 kubenswrapper[4771]: I1011 11:10:07.231185 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc4dv\" (UniqueName: \"kubernetes.io/projected/23431f0d-ef6f-4620-a467-15eda9b19df4-kube-api-access-sc4dv\") pod \"configure-network-networker-deploy-networkers-vkj22\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:07.358139 master-1 kubenswrapper[4771]: I1011 11:10:07.358049 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:10:08.028201 master-1 kubenswrapper[4771]: I1011 11:10:08.028125 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/configure-network-networker-deploy-networkers-vkj22"] Oct 11 11:10:08.037377 master-1 kubenswrapper[4771]: W1011 11:10:08.037291 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23431f0d_ef6f_4620_a467_15eda9b19df4.slice/crio-85d24faa6a4078bf4e25441dba67f8499b9fbea0924137b61c555442523d5fad WatchSource:0}: Error finding container 85d24faa6a4078bf4e25441dba67f8499b9fbea0924137b61c555442523d5fad: Status 404 returned error can't find the container with id 85d24faa6a4078bf4e25441dba67f8499b9fbea0924137b61c555442523d5fad Oct 11 11:10:08.041827 master-1 kubenswrapper[4771]: I1011 11:10:08.041796 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 11:10:08.628286 master-1 kubenswrapper[4771]: I1011 11:10:08.628244 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:10:08.728618 master-1 kubenswrapper[4771]: I1011 11:10:08.728535 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-ssh-key\") pod \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " Oct 11 11:10:08.728932 master-1 kubenswrapper[4771]: I1011 11:10:08.728679 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-inventory\") pod \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " Oct 11 11:10:08.728932 master-1 kubenswrapper[4771]: I1011 11:10:08.728747 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-bootstrap-combined-ca-bundle\") pod \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " Oct 11 11:10:08.728932 master-1 kubenswrapper[4771]: I1011 11:10:08.728847 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2frc\" (UniqueName: \"kubernetes.io/projected/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-kube-api-access-v2frc\") pod \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\" (UID: \"7bd3df1d-629a-4bbd-9ab4-7731e8928b01\") " Oct 11 11:10:08.733168 master-1 kubenswrapper[4771]: I1011 11:10:08.733107 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-bootstrap-combined-ca-bundle" (OuterVolumeSpecName: "bootstrap-combined-ca-bundle") pod "7bd3df1d-629a-4bbd-9ab4-7731e8928b01" (UID: "7bd3df1d-629a-4bbd-9ab4-7731e8928b01"). InnerVolumeSpecName "bootstrap-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:10:08.733912 master-1 kubenswrapper[4771]: I1011 11:10:08.733843 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-kube-api-access-v2frc" (OuterVolumeSpecName: "kube-api-access-v2frc") pod "7bd3df1d-629a-4bbd-9ab4-7731e8928b01" (UID: "7bd3df1d-629a-4bbd-9ab4-7731e8928b01"). InnerVolumeSpecName "kube-api-access-v2frc". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:10:08.758210 master-1 kubenswrapper[4771]: I1011 11:10:08.758132 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-inventory" (OuterVolumeSpecName: "inventory") pod "7bd3df1d-629a-4bbd-9ab4-7731e8928b01" (UID: "7bd3df1d-629a-4bbd-9ab4-7731e8928b01"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:10:08.765923 master-1 kubenswrapper[4771]: I1011 11:10:08.765829 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "7bd3df1d-629a-4bbd-9ab4-7731e8928b01" (UID: "7bd3df1d-629a-4bbd-9ab4-7731e8928b01"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:10:08.832859 master-1 kubenswrapper[4771]: I1011 11:10:08.832775 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:10:08.832859 master-1 kubenswrapper[4771]: I1011 11:10:08.832865 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:10:08.833155 master-1 kubenswrapper[4771]: I1011 11:10:08.832952 4771 reconciler_common.go:293] "Volume detached for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-bootstrap-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:10:08.833155 master-1 kubenswrapper[4771]: I1011 11:10:08.832973 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-v2frc\" (UniqueName: \"kubernetes.io/projected/7bd3df1d-629a-4bbd-9ab4-7731e8928b01-kube-api-access-v2frc\") on node \"master-1\" DevicePath \"\"" Oct 11 11:10:09.053956 master-1 kubenswrapper[4771]: I1011 11:10:09.053856 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-network-networker-deploy-networkers-vkj22" event={"ID":"23431f0d-ef6f-4620-a467-15eda9b19df4","Type":"ContainerStarted","Data":"fe2f1263dad86bbee33eaad9ff872067091a41fe3185790ad6a3291bbb8b8199"} Oct 11 11:10:09.053956 master-1 kubenswrapper[4771]: I1011 11:10:09.053943 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-network-networker-deploy-networkers-vkj22" event={"ID":"23431f0d-ef6f-4620-a467-15eda9b19df4","Type":"ContainerStarted","Data":"85d24faa6a4078bf4e25441dba67f8499b9fbea0924137b61c555442523d5fad"} Oct 11 11:10:09.056170 master-1 kubenswrapper[4771]: I1011 11:10:09.056096 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/bootstrap-dataplane-edpm-hsdgd" event={"ID":"7bd3df1d-629a-4bbd-9ab4-7731e8928b01","Type":"ContainerDied","Data":"a29718cbf38e7f25af3d27f643844ad043c8e1dc7c5a8dbda4e222f2e172deb6"} Oct 11 11:10:09.056170 master-1 kubenswrapper[4771]: I1011 11:10:09.056168 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a29718cbf38e7f25af3d27f643844ad043c8e1dc7c5a8dbda4e222f2e172deb6" Oct 11 11:10:09.056442 master-1 kubenswrapper[4771]: I1011 11:10:09.056212 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/bootstrap-dataplane-edpm-hsdgd" Oct 11 11:10:09.164257 master-1 kubenswrapper[4771]: I1011 11:10:09.164077 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/configure-network-networker-deploy-networkers-vkj22" podStartSLOduration=2.727833389 podStartE2EDuration="3.164051572s" podCreationTimestamp="2025-10-11 11:10:06 +0000 UTC" firstStartedPulling="2025-10-11 11:10:08.041725674 +0000 UTC m=+2640.015952115" lastFinishedPulling="2025-10-11 11:10:08.477943817 +0000 UTC m=+2640.452170298" observedRunningTime="2025-10-11 11:10:09.15942073 +0000 UTC m=+2641.133647261" watchObservedRunningTime="2025-10-11 11:10:09.164051572 +0000 UTC m=+2641.138278043" Oct 11 11:10:09.212546 master-1 kubenswrapper[4771]: I1011 11:10:09.212463 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/configure-network-dataplane-edpm-8f9hl"] Oct 11 11:10:09.212982 master-1 kubenswrapper[4771]: E1011 11:10:09.212943 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="7bd3df1d-629a-4bbd-9ab4-7731e8928b01" containerName="bootstrap-dataplane-edpm" Oct 11 11:10:09.212982 master-1 kubenswrapper[4771]: I1011 11:10:09.212966 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd3df1d-629a-4bbd-9ab4-7731e8928b01" containerName="bootstrap-dataplane-edpm" Oct 11 11:10:09.213239 master-1 kubenswrapper[4771]: I1011 11:10:09.213187 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="7bd3df1d-629a-4bbd-9ab4-7731e8928b01" containerName="bootstrap-dataplane-edpm" Oct 11 11:10:09.214276 master-1 kubenswrapper[4771]: I1011 11:10:09.214236 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.218336 master-1 kubenswrapper[4771]: I1011 11:10:09.218264 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:10:09.229109 master-1 kubenswrapper[4771]: I1011 11:10:09.229017 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/configure-network-dataplane-edpm-8f9hl"] Oct 11 11:10:09.346183 master-1 kubenswrapper[4771]: I1011 11:10:09.345982 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-inventory\") pod \"configure-network-dataplane-edpm-8f9hl\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.346183 master-1 kubenswrapper[4771]: I1011 11:10:09.346072 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-ssh-key\") pod \"configure-network-dataplane-edpm-8f9hl\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.346676 master-1 kubenswrapper[4771]: I1011 11:10:09.346340 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mdxh\" (UniqueName: \"kubernetes.io/projected/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-kube-api-access-9mdxh\") pod \"configure-network-dataplane-edpm-8f9hl\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.450551 master-1 kubenswrapper[4771]: I1011 11:10:09.450486 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-ssh-key\") pod \"configure-network-dataplane-edpm-8f9hl\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.450903 master-1 kubenswrapper[4771]: I1011 11:10:09.450631 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdxh\" (UniqueName: \"kubernetes.io/projected/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-kube-api-access-9mdxh\") pod \"configure-network-dataplane-edpm-8f9hl\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.450903 master-1 kubenswrapper[4771]: I1011 11:10:09.450841 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-inventory\") pod \"configure-network-dataplane-edpm-8f9hl\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.455766 master-1 kubenswrapper[4771]: I1011 11:10:09.455698 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-ssh-key\") pod \"configure-network-dataplane-edpm-8f9hl\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.456777 master-1 kubenswrapper[4771]: I1011 11:10:09.456714 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-inventory\") pod \"configure-network-dataplane-edpm-8f9hl\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.472880 master-1 kubenswrapper[4771]: I1011 11:10:09.472816 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdxh\" (UniqueName: \"kubernetes.io/projected/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-kube-api-access-9mdxh\") pod \"configure-network-dataplane-edpm-8f9hl\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:09.603408 master-1 kubenswrapper[4771]: I1011 11:10:09.603179 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:10:10.229765 master-1 kubenswrapper[4771]: I1011 11:10:10.225951 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/configure-network-dataplane-edpm-8f9hl"] Oct 11 11:10:11.082406 master-1 kubenswrapper[4771]: I1011 11:10:11.082230 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-network-dataplane-edpm-8f9hl" event={"ID":"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5","Type":"ContainerStarted","Data":"2cb0bad499f000cfdcc63010f00970a3004b6dc3b5dc657aef1ca64f72cd76cb"} Oct 11 11:10:12.097778 master-1 kubenswrapper[4771]: I1011 11:10:12.097650 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-network-dataplane-edpm-8f9hl" event={"ID":"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5","Type":"ContainerStarted","Data":"747d7c436af7449ca66e16a5f43141089dad6007d37b198bae14f81f9e107bb5"} Oct 11 11:10:12.227118 master-1 kubenswrapper[4771]: I1011 11:10:12.226996 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/configure-network-dataplane-edpm-8f9hl" podStartSLOduration=2.580016283 podStartE2EDuration="3.226961586s" podCreationTimestamp="2025-10-11 11:10:09 +0000 UTC" firstStartedPulling="2025-10-11 11:10:10.228578077 +0000 UTC m=+2642.202804508" lastFinishedPulling="2025-10-11 11:10:10.87552337 +0000 UTC m=+2642.849749811" observedRunningTime="2025-10-11 11:10:12.21767496 +0000 UTC m=+2644.191901451" watchObservedRunningTime="2025-10-11 11:10:12.226961586 +0000 UTC m=+2644.201188037" Oct 11 11:11:18.460585 master-1 kubenswrapper[4771]: I1011 11:11:18.460503 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-marketplace-sgbq8"] Oct 11 11:11:18.466449 master-1 kubenswrapper[4771]: I1011 11:11:18.462892 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.526169 master-1 kubenswrapper[4771]: I1011 11:11:18.526066 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-sgbq8"] Oct 11 11:11:18.551917 master-1 kubenswrapper[4771]: I1011 11:11:18.551844 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-catalog-content\") pod \"redhat-marketplace-sgbq8\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.552167 master-1 kubenswrapper[4771]: I1011 11:11:18.552036 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92h48\" (UniqueName: \"kubernetes.io/projected/bc2314e8-dacc-4d65-b836-58b7453a60fa-kube-api-access-92h48\") pod \"redhat-marketplace-sgbq8\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.552293 master-1 kubenswrapper[4771]: I1011 11:11:18.552259 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-utilities\") pod \"redhat-marketplace-sgbq8\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.654611 master-1 kubenswrapper[4771]: I1011 11:11:18.654480 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-catalog-content\") pod \"redhat-marketplace-sgbq8\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.655532 master-1 kubenswrapper[4771]: I1011 11:11:18.655486 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-92h48\" (UniqueName: \"kubernetes.io/projected/bc2314e8-dacc-4d65-b836-58b7453a60fa-kube-api-access-92h48\") pod \"redhat-marketplace-sgbq8\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.655692 master-1 kubenswrapper[4771]: I1011 11:11:18.655649 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-utilities\") pod \"redhat-marketplace-sgbq8\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.656148 master-1 kubenswrapper[4771]: I1011 11:11:18.656105 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-utilities\") pod \"redhat-marketplace-sgbq8\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.656569 master-1 kubenswrapper[4771]: I1011 11:11:18.656526 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-catalog-content\") pod \"redhat-marketplace-sgbq8\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.679680 master-1 kubenswrapper[4771]: I1011 11:11:18.679608 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-92h48\" (UniqueName: \"kubernetes.io/projected/bc2314e8-dacc-4d65-b836-58b7453a60fa-kube-api-access-92h48\") pod \"redhat-marketplace-sgbq8\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:18.847606 master-1 kubenswrapper[4771]: I1011 11:11:18.847513 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:19.304993 master-1 kubenswrapper[4771]: W1011 11:11:19.304901 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2314e8_dacc_4d65_b836_58b7453a60fa.slice/crio-53a505e30cbb5b3189f3cc84cceeaa763702420367f06fbafef2bbdae9c2391e WatchSource:0}: Error finding container 53a505e30cbb5b3189f3cc84cceeaa763702420367f06fbafef2bbdae9c2391e: Status 404 returned error can't find the container with id 53a505e30cbb5b3189f3cc84cceeaa763702420367f06fbafef2bbdae9c2391e Oct 11 11:11:19.305349 master-1 kubenswrapper[4771]: I1011 11:11:19.305251 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-sgbq8"] Oct 11 11:11:19.810975 master-1 kubenswrapper[4771]: I1011 11:11:19.810849 4771 generic.go:334] "Generic (PLEG): container finished" podID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerID="e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369" exitCode=0 Oct 11 11:11:19.810975 master-1 kubenswrapper[4771]: I1011 11:11:19.810947 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-sgbq8" event={"ID":"bc2314e8-dacc-4d65-b836-58b7453a60fa","Type":"ContainerDied","Data":"e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369"} Oct 11 11:11:19.812102 master-1 kubenswrapper[4771]: I1011 11:11:19.811002 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-sgbq8" event={"ID":"bc2314e8-dacc-4d65-b836-58b7453a60fa","Type":"ContainerStarted","Data":"53a505e30cbb5b3189f3cc84cceeaa763702420367f06fbafef2bbdae9c2391e"} Oct 11 11:11:21.835594 master-1 kubenswrapper[4771]: I1011 11:11:21.835504 4771 generic.go:334] "Generic (PLEG): container finished" podID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerID="f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d" exitCode=0 Oct 11 11:11:21.836505 master-1 kubenswrapper[4771]: I1011 11:11:21.835584 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-sgbq8" event={"ID":"bc2314e8-dacc-4d65-b836-58b7453a60fa","Type":"ContainerDied","Data":"f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d"} Oct 11 11:11:22.852305 master-1 kubenswrapper[4771]: I1011 11:11:22.852210 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-sgbq8" event={"ID":"bc2314e8-dacc-4d65-b836-58b7453a60fa","Type":"ContainerStarted","Data":"611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a"} Oct 11 11:11:22.894995 master-1 kubenswrapper[4771]: I1011 11:11:22.894875 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-marketplace-sgbq8" podStartSLOduration=2.459694403 podStartE2EDuration="4.894848741s" podCreationTimestamp="2025-10-11 11:11:18 +0000 UTC" firstStartedPulling="2025-10-11 11:11:19.813475177 +0000 UTC m=+2711.787701618" lastFinishedPulling="2025-10-11 11:11:22.248629475 +0000 UTC m=+2714.222855956" observedRunningTime="2025-10-11 11:11:22.881385495 +0000 UTC m=+2714.855612026" watchObservedRunningTime="2025-10-11 11:11:22.894848741 +0000 UTC m=+2714.869075212" Oct 11 11:11:28.848761 master-1 kubenswrapper[4771]: I1011 11:11:28.848688 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:28.848761 master-1 kubenswrapper[4771]: I1011 11:11:28.848732 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:28.900927 master-1 kubenswrapper[4771]: I1011 11:11:28.900880 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:28.980875 master-1 kubenswrapper[4771]: I1011 11:11:28.980794 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:29.159503 master-1 kubenswrapper[4771]: I1011 11:11:29.159091 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-sgbq8"] Oct 11 11:11:29.248382 master-1 kubenswrapper[4771]: E1011 11:11:29.248268 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb37e2544_0a7f_44ec_9e4a_f2736c93b9f5.slice/crio-747d7c436af7449ca66e16a5f43141089dad6007d37b198bae14f81f9e107bb5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb37e2544_0a7f_44ec_9e4a_f2736c93b9f5.slice/crio-conmon-747d7c436af7449ca66e16a5f43141089dad6007d37b198bae14f81f9e107bb5.scope\": RecentStats: unable to find data in memory cache]" Oct 11 11:11:29.248784 master-1 kubenswrapper[4771]: E1011 11:11:29.248562 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb37e2544_0a7f_44ec_9e4a_f2736c93b9f5.slice/crio-conmon-747d7c436af7449ca66e16a5f43141089dad6007d37b198bae14f81f9e107bb5.scope\": RecentStats: unable to find data in memory cache]" Oct 11 11:11:29.925169 master-1 kubenswrapper[4771]: I1011 11:11:29.925075 4771 generic.go:334] "Generic (PLEG): container finished" podID="b37e2544-0a7f-44ec-9e4a-f2736c93b9f5" containerID="747d7c436af7449ca66e16a5f43141089dad6007d37b198bae14f81f9e107bb5" exitCode=0 Oct 11 11:11:29.926546 master-1 kubenswrapper[4771]: I1011 11:11:29.926344 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-network-dataplane-edpm-8f9hl" event={"ID":"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5","Type":"ContainerDied","Data":"747d7c436af7449ca66e16a5f43141089dad6007d37b198bae14f81f9e107bb5"} Oct 11 11:11:30.934610 master-1 kubenswrapper[4771]: I1011 11:11:30.934502 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-marketplace-sgbq8" podUID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerName="registry-server" containerID="cri-o://611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a" gracePeriod=2 Oct 11 11:11:31.615188 master-1 kubenswrapper[4771]: I1011 11:11:31.615154 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:11:31.716715 master-1 kubenswrapper[4771]: I1011 11:11:31.716660 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:31.776466 master-1 kubenswrapper[4771]: I1011 11:11:31.776405 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-ssh-key\") pod \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " Oct 11 11:11:31.776754 master-1 kubenswrapper[4771]: I1011 11:11:31.776570 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-inventory\") pod \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " Oct 11 11:11:31.776754 master-1 kubenswrapper[4771]: I1011 11:11:31.776670 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mdxh\" (UniqueName: \"kubernetes.io/projected/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-kube-api-access-9mdxh\") pod \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\" (UID: \"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5\") " Oct 11 11:11:31.782349 master-1 kubenswrapper[4771]: I1011 11:11:31.782230 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-kube-api-access-9mdxh" (OuterVolumeSpecName: "kube-api-access-9mdxh") pod "b37e2544-0a7f-44ec-9e4a-f2736c93b9f5" (UID: "b37e2544-0a7f-44ec-9e4a-f2736c93b9f5"). InnerVolumeSpecName "kube-api-access-9mdxh". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:11:31.801768 master-1 kubenswrapper[4771]: I1011 11:11:31.801712 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "b37e2544-0a7f-44ec-9e4a-f2736c93b9f5" (UID: "b37e2544-0a7f-44ec-9e4a-f2736c93b9f5"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:11:31.818741 master-1 kubenswrapper[4771]: I1011 11:11:31.818584 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-inventory" (OuterVolumeSpecName: "inventory") pod "b37e2544-0a7f-44ec-9e4a-f2736c93b9f5" (UID: "b37e2544-0a7f-44ec-9e4a-f2736c93b9f5"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:11:31.878105 master-1 kubenswrapper[4771]: I1011 11:11:31.878049 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-catalog-content\") pod \"bc2314e8-dacc-4d65-b836-58b7453a60fa\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " Oct 11 11:11:31.878505 master-1 kubenswrapper[4771]: I1011 11:11:31.878285 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-utilities\") pod \"bc2314e8-dacc-4d65-b836-58b7453a60fa\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " Oct 11 11:11:31.878505 master-1 kubenswrapper[4771]: I1011 11:11:31.878390 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92h48\" (UniqueName: \"kubernetes.io/projected/bc2314e8-dacc-4d65-b836-58b7453a60fa-kube-api-access-92h48\") pod \"bc2314e8-dacc-4d65-b836-58b7453a60fa\" (UID: \"bc2314e8-dacc-4d65-b836-58b7453a60fa\") " Oct 11 11:11:31.879181 master-1 kubenswrapper[4771]: I1011 11:11:31.878803 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:31.879181 master-1 kubenswrapper[4771]: I1011 11:11:31.878821 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:31.879181 master-1 kubenswrapper[4771]: I1011 11:11:31.878831 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-9mdxh\" (UniqueName: \"kubernetes.io/projected/b37e2544-0a7f-44ec-9e4a-f2736c93b9f5-kube-api-access-9mdxh\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:31.879944 master-1 kubenswrapper[4771]: I1011 11:11:31.879871 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-utilities" (OuterVolumeSpecName: "utilities") pod "bc2314e8-dacc-4d65-b836-58b7453a60fa" (UID: "bc2314e8-dacc-4d65-b836-58b7453a60fa"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:11:31.884659 master-1 kubenswrapper[4771]: I1011 11:11:31.883586 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2314e8-dacc-4d65-b836-58b7453a60fa-kube-api-access-92h48" (OuterVolumeSpecName: "kube-api-access-92h48") pod "bc2314e8-dacc-4d65-b836-58b7453a60fa" (UID: "bc2314e8-dacc-4d65-b836-58b7453a60fa"). InnerVolumeSpecName "kube-api-access-92h48". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:11:31.916292 master-1 kubenswrapper[4771]: I1011 11:11:31.916183 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "bc2314e8-dacc-4d65-b836-58b7453a60fa" (UID: "bc2314e8-dacc-4d65-b836-58b7453a60fa"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:11:31.944916 master-1 kubenswrapper[4771]: I1011 11:11:31.944389 4771 generic.go:334] "Generic (PLEG): container finished" podID="23431f0d-ef6f-4620-a467-15eda9b19df4" containerID="fe2f1263dad86bbee33eaad9ff872067091a41fe3185790ad6a3291bbb8b8199" exitCode=0 Oct 11 11:11:31.944916 master-1 kubenswrapper[4771]: I1011 11:11:31.944431 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-network-networker-deploy-networkers-vkj22" event={"ID":"23431f0d-ef6f-4620-a467-15eda9b19df4","Type":"ContainerDied","Data":"fe2f1263dad86bbee33eaad9ff872067091a41fe3185790ad6a3291bbb8b8199"} Oct 11 11:11:31.946877 master-1 kubenswrapper[4771]: I1011 11:11:31.946835 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-network-dataplane-edpm-8f9hl" event={"ID":"b37e2544-0a7f-44ec-9e4a-f2736c93b9f5","Type":"ContainerDied","Data":"2cb0bad499f000cfdcc63010f00970a3004b6dc3b5dc657aef1ca64f72cd76cb"} Oct 11 11:11:31.946967 master-1 kubenswrapper[4771]: I1011 11:11:31.946880 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cb0bad499f000cfdcc63010f00970a3004b6dc3b5dc657aef1ca64f72cd76cb" Oct 11 11:11:31.946967 master-1 kubenswrapper[4771]: I1011 11:11:31.946884 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/configure-network-dataplane-edpm-8f9hl" Oct 11 11:11:31.950375 master-1 kubenswrapper[4771]: I1011 11:11:31.950278 4771 generic.go:334] "Generic (PLEG): container finished" podID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerID="611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a" exitCode=0 Oct 11 11:11:31.950375 master-1 kubenswrapper[4771]: I1011 11:11:31.950347 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-sgbq8" Oct 11 11:11:31.950504 master-1 kubenswrapper[4771]: I1011 11:11:31.950384 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-sgbq8" event={"ID":"bc2314e8-dacc-4d65-b836-58b7453a60fa","Type":"ContainerDied","Data":"611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a"} Oct 11 11:11:31.950567 master-1 kubenswrapper[4771]: I1011 11:11:31.950538 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-sgbq8" event={"ID":"bc2314e8-dacc-4d65-b836-58b7453a60fa","Type":"ContainerDied","Data":"53a505e30cbb5b3189f3cc84cceeaa763702420367f06fbafef2bbdae9c2391e"} Oct 11 11:11:31.950623 master-1 kubenswrapper[4771]: I1011 11:11:31.950587 4771 scope.go:117] "RemoveContainer" containerID="611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a" Oct 11 11:11:31.978789 master-1 kubenswrapper[4771]: I1011 11:11:31.978616 4771 scope.go:117] "RemoveContainer" containerID="f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d" Oct 11 11:11:31.982048 master-1 kubenswrapper[4771]: I1011 11:11:31.981998 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-92h48\" (UniqueName: \"kubernetes.io/projected/bc2314e8-dacc-4d65-b836-58b7453a60fa-kube-api-access-92h48\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:31.982105 master-1 kubenswrapper[4771]: I1011 11:11:31.982055 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:31.982105 master-1 kubenswrapper[4771]: I1011 11:11:31.982074 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/bc2314e8-dacc-4d65-b836-58b7453a60fa-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:32.029732 master-1 kubenswrapper[4771]: I1011 11:11:32.029651 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-sgbq8"] Oct 11 11:11:32.032282 master-1 kubenswrapper[4771]: I1011 11:11:32.032236 4771 scope.go:117] "RemoveContainer" containerID="e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369" Oct 11 11:11:32.037875 master-1 kubenswrapper[4771]: I1011 11:11:32.037834 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-marketplace-sgbq8"] Oct 11 11:11:32.059085 master-1 kubenswrapper[4771]: I1011 11:11:32.059046 4771 scope.go:117] "RemoveContainer" containerID="611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a" Oct 11 11:11:32.059716 master-1 kubenswrapper[4771]: E1011 11:11:32.059672 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a\": container with ID starting with 611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a not found: ID does not exist" containerID="611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a" Oct 11 11:11:32.059885 master-1 kubenswrapper[4771]: I1011 11:11:32.059711 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a"} err="failed to get container status \"611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a\": rpc error: code = NotFound desc = could not find container \"611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a\": container with ID starting with 611612c9745148a9971c30e141372de2a008a018cd29e07a94c85a40014e378a not found: ID does not exist" Oct 11 11:11:32.059885 master-1 kubenswrapper[4771]: I1011 11:11:32.059779 4771 scope.go:117] "RemoveContainer" containerID="f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d" Oct 11 11:11:32.060443 master-1 kubenswrapper[4771]: E1011 11:11:32.060401 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d\": container with ID starting with f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d not found: ID does not exist" containerID="f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d" Oct 11 11:11:32.060515 master-1 kubenswrapper[4771]: I1011 11:11:32.060442 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d"} err="failed to get container status \"f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d\": rpc error: code = NotFound desc = could not find container \"f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d\": container with ID starting with f5f44a51efa64fb057e1bc45ff72f7b6e206ce4ce789d4bdd15bfeada257f07d not found: ID does not exist" Oct 11 11:11:32.060515 master-1 kubenswrapper[4771]: I1011 11:11:32.060471 4771 scope.go:117] "RemoveContainer" containerID="e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369" Oct 11 11:11:32.060849 master-1 kubenswrapper[4771]: E1011 11:11:32.060824 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369\": container with ID starting with e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369 not found: ID does not exist" containerID="e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369" Oct 11 11:11:32.060922 master-1 kubenswrapper[4771]: I1011 11:11:32.060854 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369"} err="failed to get container status \"e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369\": rpc error: code = NotFound desc = could not find container \"e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369\": container with ID starting with e5ed3ec47283fd94f26c6b7d58982128685eab7226f768e7fba4c17ae866e369 not found: ID does not exist" Oct 11 11:11:32.456182 master-1 kubenswrapper[4771]: I1011 11:11:32.455975 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc2314e8-dacc-4d65-b836-58b7453a60fa" path="/var/lib/kubelet/pods/bc2314e8-dacc-4d65-b836-58b7453a60fa/volumes" Oct 11 11:11:33.559972 master-1 kubenswrapper[4771]: I1011 11:11:33.559830 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:11:33.733195 master-1 kubenswrapper[4771]: I1011 11:11:33.733018 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-inventory\") pod \"23431f0d-ef6f-4620-a467-15eda9b19df4\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " Oct 11 11:11:33.733608 master-1 kubenswrapper[4771]: I1011 11:11:33.733245 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-ssh-key\") pod \"23431f0d-ef6f-4620-a467-15eda9b19df4\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " Oct 11 11:11:33.733677 master-1 kubenswrapper[4771]: I1011 11:11:33.733636 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc4dv\" (UniqueName: \"kubernetes.io/projected/23431f0d-ef6f-4620-a467-15eda9b19df4-kube-api-access-sc4dv\") pod \"23431f0d-ef6f-4620-a467-15eda9b19df4\" (UID: \"23431f0d-ef6f-4620-a467-15eda9b19df4\") " Oct 11 11:11:33.743992 master-1 kubenswrapper[4771]: I1011 11:11:33.743896 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23431f0d-ef6f-4620-a467-15eda9b19df4-kube-api-access-sc4dv" (OuterVolumeSpecName: "kube-api-access-sc4dv") pod "23431f0d-ef6f-4620-a467-15eda9b19df4" (UID: "23431f0d-ef6f-4620-a467-15eda9b19df4"). InnerVolumeSpecName "kube-api-access-sc4dv". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:11:33.778089 master-1 kubenswrapper[4771]: I1011 11:11:33.777988 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-inventory" (OuterVolumeSpecName: "inventory") pod "23431f0d-ef6f-4620-a467-15eda9b19df4" (UID: "23431f0d-ef6f-4620-a467-15eda9b19df4"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:11:33.783281 master-1 kubenswrapper[4771]: I1011 11:11:33.783206 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "23431f0d-ef6f-4620-a467-15eda9b19df4" (UID: "23431f0d-ef6f-4620-a467-15eda9b19df4"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:11:33.836940 master-1 kubenswrapper[4771]: I1011 11:11:33.836867 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:33.836940 master-1 kubenswrapper[4771]: I1011 11:11:33.836940 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/23431f0d-ef6f-4620-a467-15eda9b19df4-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:33.837079 master-1 kubenswrapper[4771]: I1011 11:11:33.836962 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-sc4dv\" (UniqueName: \"kubernetes.io/projected/23431f0d-ef6f-4620-a467-15eda9b19df4-kube-api-access-sc4dv\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:33.977414 master-1 kubenswrapper[4771]: I1011 11:11:33.977190 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-network-networker-deploy-networkers-vkj22" event={"ID":"23431f0d-ef6f-4620-a467-15eda9b19df4","Type":"ContainerDied","Data":"85d24faa6a4078bf4e25441dba67f8499b9fbea0924137b61c555442523d5fad"} Oct 11 11:11:33.977414 master-1 kubenswrapper[4771]: I1011 11:11:33.977254 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/configure-network-networker-deploy-networkers-vkj22" Oct 11 11:11:33.977414 master-1 kubenswrapper[4771]: I1011 11:11:33.977268 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d24faa6a4078bf4e25441dba67f8499b9fbea0924137b61c555442523d5fad" Oct 11 11:11:34.287468 master-1 kubenswrapper[4771]: I1011 11:11:34.286594 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/validate-network-networker-deploy-networkers-m7845"] Oct 11 11:11:34.287468 master-1 kubenswrapper[4771]: E1011 11:11:34.287066 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerName="extract-utilities" Oct 11 11:11:34.287468 master-1 kubenswrapper[4771]: I1011 11:11:34.287085 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerName="extract-utilities" Oct 11 11:11:34.287468 master-1 kubenswrapper[4771]: E1011 11:11:34.287105 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="b37e2544-0a7f-44ec-9e4a-f2736c93b9f5" containerName="configure-network-dataplane-edpm" Oct 11 11:11:34.287468 master-1 kubenswrapper[4771]: I1011 11:11:34.287471 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37e2544-0a7f-44ec-9e4a-f2736c93b9f5" containerName="configure-network-dataplane-edpm" Oct 11 11:11:34.287896 master-1 kubenswrapper[4771]: E1011 11:11:34.287497 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="23431f0d-ef6f-4620-a467-15eda9b19df4" containerName="configure-network-networker-deploy-networkers" Oct 11 11:11:34.287896 master-1 kubenswrapper[4771]: I1011 11:11:34.287509 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="23431f0d-ef6f-4620-a467-15eda9b19df4" containerName="configure-network-networker-deploy-networkers" Oct 11 11:11:34.287896 master-1 kubenswrapper[4771]: E1011 11:11:34.287522 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerName="registry-server" Oct 11 11:11:34.287896 master-1 kubenswrapper[4771]: I1011 11:11:34.287529 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerName="registry-server" Oct 11 11:11:34.287896 master-1 kubenswrapper[4771]: E1011 11:11:34.287552 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerName="extract-content" Oct 11 11:11:34.287896 master-1 kubenswrapper[4771]: I1011 11:11:34.287559 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerName="extract-content" Oct 11 11:11:34.288168 master-1 kubenswrapper[4771]: I1011 11:11:34.287960 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="bc2314e8-dacc-4d65-b836-58b7453a60fa" containerName="registry-server" Oct 11 11:11:34.288168 master-1 kubenswrapper[4771]: I1011 11:11:34.287999 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="23431f0d-ef6f-4620-a467-15eda9b19df4" containerName="configure-network-networker-deploy-networkers" Oct 11 11:11:34.288168 master-1 kubenswrapper[4771]: I1011 11:11:34.288019 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="b37e2544-0a7f-44ec-9e4a-f2736c93b9f5" containerName="configure-network-dataplane-edpm" Oct 11 11:11:34.291044 master-1 kubenswrapper[4771]: I1011 11:11:34.290983 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.294979 master-1 kubenswrapper[4771]: I1011 11:11:34.294931 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-networkers" Oct 11 11:11:34.295813 master-1 kubenswrapper[4771]: I1011 11:11:34.295605 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:11:34.296347 master-1 kubenswrapper[4771]: I1011 11:11:34.296250 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:11:34.324422 master-1 kubenswrapper[4771]: I1011 11:11:34.315091 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/validate-network-networker-deploy-networkers-m7845"] Oct 11 11:11:34.349684 master-1 kubenswrapper[4771]: I1011 11:11:34.349593 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-inventory\") pod \"validate-network-networker-deploy-networkers-m7845\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.349920 master-1 kubenswrapper[4771]: I1011 11:11:34.349699 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7qp\" (UniqueName: \"kubernetes.io/projected/32524a1a-75d4-47d4-81e1-ef8562425eb3-kube-api-access-2w7qp\") pod \"validate-network-networker-deploy-networkers-m7845\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.350125 master-1 kubenswrapper[4771]: I1011 11:11:34.350062 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-ssh-key\") pod \"validate-network-networker-deploy-networkers-m7845\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.462962 master-1 kubenswrapper[4771]: I1011 11:11:34.462862 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-ssh-key\") pod \"validate-network-networker-deploy-networkers-m7845\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.462962 master-1 kubenswrapper[4771]: I1011 11:11:34.462966 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-inventory\") pod \"validate-network-networker-deploy-networkers-m7845\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.463272 master-1 kubenswrapper[4771]: I1011 11:11:34.463022 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7qp\" (UniqueName: \"kubernetes.io/projected/32524a1a-75d4-47d4-81e1-ef8562425eb3-kube-api-access-2w7qp\") pod \"validate-network-networker-deploy-networkers-m7845\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.468971 master-1 kubenswrapper[4771]: I1011 11:11:34.468904 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-ssh-key\") pod \"validate-network-networker-deploy-networkers-m7845\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.469993 master-1 kubenswrapper[4771]: I1011 11:11:34.469948 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-inventory\") pod \"validate-network-networker-deploy-networkers-m7845\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.488130 master-1 kubenswrapper[4771]: I1011 11:11:34.488047 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7qp\" (UniqueName: \"kubernetes.io/projected/32524a1a-75d4-47d4-81e1-ef8562425eb3-kube-api-access-2w7qp\") pod \"validate-network-networker-deploy-networkers-m7845\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:34.626634 master-1 kubenswrapper[4771]: I1011 11:11:34.626533 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:35.263603 master-1 kubenswrapper[4771]: I1011 11:11:35.263517 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/validate-network-networker-deploy-networkers-m7845"] Oct 11 11:11:35.272454 master-1 kubenswrapper[4771]: W1011 11:11:35.272334 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32524a1a_75d4_47d4_81e1_ef8562425eb3.slice/crio-8102d15a2bc375478cb70b78596a84085e5e1f3d04840dcc27705e78d0412342 WatchSource:0}: Error finding container 8102d15a2bc375478cb70b78596a84085e5e1f3d04840dcc27705e78d0412342: Status 404 returned error can't find the container with id 8102d15a2bc375478cb70b78596a84085e5e1f3d04840dcc27705e78d0412342 Oct 11 11:11:36.010450 master-1 kubenswrapper[4771]: I1011 11:11:36.010248 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/validate-network-networker-deploy-networkers-m7845" event={"ID":"32524a1a-75d4-47d4-81e1-ef8562425eb3","Type":"ContainerStarted","Data":"29a03a08ba4a91706056a19084966a858f7a063a71bad2d0e4a86c00ea696baa"} Oct 11 11:11:36.011330 master-1 kubenswrapper[4771]: I1011 11:11:36.011293 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/validate-network-networker-deploy-networkers-m7845" event={"ID":"32524a1a-75d4-47d4-81e1-ef8562425eb3","Type":"ContainerStarted","Data":"8102d15a2bc375478cb70b78596a84085e5e1f3d04840dcc27705e78d0412342"} Oct 11 11:11:36.035820 master-1 kubenswrapper[4771]: I1011 11:11:36.035687 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/validate-network-networker-deploy-networkers-m7845" podStartSLOduration=1.609112871 podStartE2EDuration="2.035654567s" podCreationTimestamp="2025-10-11 11:11:34 +0000 UTC" firstStartedPulling="2025-10-11 11:11:35.277304189 +0000 UTC m=+2727.251530640" lastFinishedPulling="2025-10-11 11:11:35.703845855 +0000 UTC m=+2727.678072336" observedRunningTime="2025-10-11 11:11:36.03330206 +0000 UTC m=+2728.007528541" watchObservedRunningTime="2025-10-11 11:11:36.035654567 +0000 UTC m=+2728.009881048" Oct 11 11:11:42.083802 master-1 kubenswrapper[4771]: I1011 11:11:42.083754 4771 generic.go:334] "Generic (PLEG): container finished" podID="32524a1a-75d4-47d4-81e1-ef8562425eb3" containerID="29a03a08ba4a91706056a19084966a858f7a063a71bad2d0e4a86c00ea696baa" exitCode=0 Oct 11 11:11:42.084415 master-1 kubenswrapper[4771]: I1011 11:11:42.083888 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/validate-network-networker-deploy-networkers-m7845" event={"ID":"32524a1a-75d4-47d4-81e1-ef8562425eb3","Type":"ContainerDied","Data":"29a03a08ba4a91706056a19084966a858f7a063a71bad2d0e4a86c00ea696baa"} Oct 11 11:11:43.749274 master-1 kubenswrapper[4771]: I1011 11:11:43.749200 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:43.799408 master-1 kubenswrapper[4771]: I1011 11:11:43.799330 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-inventory\") pod \"32524a1a-75d4-47d4-81e1-ef8562425eb3\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " Oct 11 11:11:43.799979 master-1 kubenswrapper[4771]: I1011 11:11:43.799449 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w7qp\" (UniqueName: \"kubernetes.io/projected/32524a1a-75d4-47d4-81e1-ef8562425eb3-kube-api-access-2w7qp\") pod \"32524a1a-75d4-47d4-81e1-ef8562425eb3\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " Oct 11 11:11:43.799979 master-1 kubenswrapper[4771]: I1011 11:11:43.799610 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-ssh-key\") pod \"32524a1a-75d4-47d4-81e1-ef8562425eb3\" (UID: \"32524a1a-75d4-47d4-81e1-ef8562425eb3\") " Oct 11 11:11:43.808039 master-1 kubenswrapper[4771]: I1011 11:11:43.807956 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32524a1a-75d4-47d4-81e1-ef8562425eb3-kube-api-access-2w7qp" (OuterVolumeSpecName: "kube-api-access-2w7qp") pod "32524a1a-75d4-47d4-81e1-ef8562425eb3" (UID: "32524a1a-75d4-47d4-81e1-ef8562425eb3"). InnerVolumeSpecName "kube-api-access-2w7qp". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:11:43.827553 master-1 kubenswrapper[4771]: I1011 11:11:43.827469 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "32524a1a-75d4-47d4-81e1-ef8562425eb3" (UID: "32524a1a-75d4-47d4-81e1-ef8562425eb3"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:11:43.868167 master-1 kubenswrapper[4771]: I1011 11:11:43.868106 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-inventory" (OuterVolumeSpecName: "inventory") pod "32524a1a-75d4-47d4-81e1-ef8562425eb3" (UID: "32524a1a-75d4-47d4-81e1-ef8562425eb3"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:11:43.901181 master-1 kubenswrapper[4771]: I1011 11:11:43.901117 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:43.901181 master-1 kubenswrapper[4771]: I1011 11:11:43.901170 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-2w7qp\" (UniqueName: \"kubernetes.io/projected/32524a1a-75d4-47d4-81e1-ef8562425eb3-kube-api-access-2w7qp\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:43.901412 master-1 kubenswrapper[4771]: I1011 11:11:43.901189 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/32524a1a-75d4-47d4-81e1-ef8562425eb3-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:11:44.107284 master-1 kubenswrapper[4771]: I1011 11:11:44.107207 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/validate-network-networker-deploy-networkers-m7845" event={"ID":"32524a1a-75d4-47d4-81e1-ef8562425eb3","Type":"ContainerDied","Data":"8102d15a2bc375478cb70b78596a84085e5e1f3d04840dcc27705e78d0412342"} Oct 11 11:11:44.107559 master-1 kubenswrapper[4771]: I1011 11:11:44.107305 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8102d15a2bc375478cb70b78596a84085e5e1f3d04840dcc27705e78d0412342" Oct 11 11:11:44.107559 master-1 kubenswrapper[4771]: I1011 11:11:44.107313 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/validate-network-networker-deploy-networkers-m7845" Oct 11 11:11:44.227790 master-1 kubenswrapper[4771]: I1011 11:11:44.227704 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/install-os-networker-deploy-networkers-6txzc"] Oct 11 11:11:44.228349 master-1 kubenswrapper[4771]: E1011 11:11:44.228303 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="32524a1a-75d4-47d4-81e1-ef8562425eb3" containerName="validate-network-networker-deploy-networkers" Oct 11 11:11:44.228349 master-1 kubenswrapper[4771]: I1011 11:11:44.228342 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="32524a1a-75d4-47d4-81e1-ef8562425eb3" containerName="validate-network-networker-deploy-networkers" Oct 11 11:11:44.228718 master-1 kubenswrapper[4771]: I1011 11:11:44.228676 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="32524a1a-75d4-47d4-81e1-ef8562425eb3" containerName="validate-network-networker-deploy-networkers" Oct 11 11:11:44.232791 master-1 kubenswrapper[4771]: I1011 11:11:44.231062 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.235236 master-1 kubenswrapper[4771]: I1011 11:11:44.235128 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:11:44.235490 master-1 kubenswrapper[4771]: I1011 11:11:44.235441 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:11:44.235945 master-1 kubenswrapper[4771]: I1011 11:11:44.235902 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-networkers" Oct 11 11:11:44.289022 master-1 kubenswrapper[4771]: I1011 11:11:44.288935 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/install-os-networker-deploy-networkers-6txzc"] Oct 11 11:11:44.311852 master-1 kubenswrapper[4771]: I1011 11:11:44.311777 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-inventory\") pod \"install-os-networker-deploy-networkers-6txzc\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.312013 master-1 kubenswrapper[4771]: I1011 11:11:44.311863 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrrm\" (UniqueName: \"kubernetes.io/projected/24d0c33f-c29a-4878-a73d-4fdc54ee4754-kube-api-access-flrrm\") pod \"install-os-networker-deploy-networkers-6txzc\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.312440 master-1 kubenswrapper[4771]: I1011 11:11:44.312400 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-ssh-key\") pod \"install-os-networker-deploy-networkers-6txzc\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.414988 master-1 kubenswrapper[4771]: I1011 11:11:44.414749 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-ssh-key\") pod \"install-os-networker-deploy-networkers-6txzc\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.414988 master-1 kubenswrapper[4771]: I1011 11:11:44.414852 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-inventory\") pod \"install-os-networker-deploy-networkers-6txzc\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.414988 master-1 kubenswrapper[4771]: I1011 11:11:44.414884 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-flrrm\" (UniqueName: \"kubernetes.io/projected/24d0c33f-c29a-4878-a73d-4fdc54ee4754-kube-api-access-flrrm\") pod \"install-os-networker-deploy-networkers-6txzc\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.419165 master-1 kubenswrapper[4771]: I1011 11:11:44.419094 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-ssh-key\") pod \"install-os-networker-deploy-networkers-6txzc\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.419290 master-1 kubenswrapper[4771]: I1011 11:11:44.419230 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-inventory\") pod \"install-os-networker-deploy-networkers-6txzc\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.439836 master-1 kubenswrapper[4771]: I1011 11:11:44.439740 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrrm\" (UniqueName: \"kubernetes.io/projected/24d0c33f-c29a-4878-a73d-4fdc54ee4754-kube-api-access-flrrm\") pod \"install-os-networker-deploy-networkers-6txzc\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:44.556951 master-1 kubenswrapper[4771]: I1011 11:11:44.556712 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:11:45.186173 master-1 kubenswrapper[4771]: I1011 11:11:45.186108 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/install-os-networker-deploy-networkers-6txzc"] Oct 11 11:11:45.192271 master-1 kubenswrapper[4771]: W1011 11:11:45.192076 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24d0c33f_c29a_4878_a73d_4fdc54ee4754.slice/crio-358ed73fdc20984b748856c249dfddc8dcd94200b65b4cf2e53a249cbb6512a0 WatchSource:0}: Error finding container 358ed73fdc20984b748856c249dfddc8dcd94200b65b4cf2e53a249cbb6512a0: Status 404 returned error can't find the container with id 358ed73fdc20984b748856c249dfddc8dcd94200b65b4cf2e53a249cbb6512a0 Oct 11 11:11:46.131252 master-1 kubenswrapper[4771]: I1011 11:11:46.131126 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/install-os-networker-deploy-networkers-6txzc" event={"ID":"24d0c33f-c29a-4878-a73d-4fdc54ee4754","Type":"ContainerStarted","Data":"8dfae439c8c304c6fb6f9f04b4870358e01be13642192b2cd7e8e013d0312a0b"} Oct 11 11:11:46.131673 master-1 kubenswrapper[4771]: I1011 11:11:46.131494 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/install-os-networker-deploy-networkers-6txzc" event={"ID":"24d0c33f-c29a-4878-a73d-4fdc54ee4754","Type":"ContainerStarted","Data":"358ed73fdc20984b748856c249dfddc8dcd94200b65b4cf2e53a249cbb6512a0"} Oct 11 11:11:46.158257 master-1 kubenswrapper[4771]: I1011 11:11:46.158123 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/install-os-networker-deploy-networkers-6txzc" podStartSLOduration=1.6747316300000001 podStartE2EDuration="2.158081202s" podCreationTimestamp="2025-10-11 11:11:44 +0000 UTC" firstStartedPulling="2025-10-11 11:11:45.196444353 +0000 UTC m=+2737.170670804" lastFinishedPulling="2025-10-11 11:11:45.679793935 +0000 UTC m=+2737.654020376" observedRunningTime="2025-10-11 11:11:46.153172252 +0000 UTC m=+2738.127398753" watchObservedRunningTime="2025-10-11 11:11:46.158081202 +0000 UTC m=+2738.132307673" Oct 11 11:12:15.070312 master-1 kubenswrapper[4771]: I1011 11:12:15.069784 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/octavia-db-create-rbtpx"] Oct 11 11:12:15.084615 master-1 kubenswrapper[4771]: I1011 11:12:15.084547 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/octavia-db-create-rbtpx"] Oct 11 11:12:16.450262 master-1 kubenswrapper[4771]: I1011 11:12:16.450166 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041373ee-1533-4bc6-abd2-80d16bfa5f23" path="/var/lib/kubelet/pods/041373ee-1533-4bc6-abd2-80d16bfa5f23/volumes" Oct 11 11:12:24.872458 master-1 kubenswrapper[4771]: I1011 11:12:24.872342 4771 scope.go:117] "RemoveContainer" containerID="736c15aefe67b305735f91c4e8c2109ad242f954cfb3635af77a747443410e30" Oct 11 11:12:27.060538 master-1 kubenswrapper[4771]: I1011 11:12:27.060421 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/octavia-e4bf-account-create-m2n6j"] Oct 11 11:12:27.074415 master-1 kubenswrapper[4771]: I1011 11:12:27.074311 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/octavia-e4bf-account-create-m2n6j"] Oct 11 11:12:28.461230 master-1 kubenswrapper[4771]: I1011 11:12:28.461140 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba52ba8-24c1-4b0c-83cb-6837e2353fa8" path="/var/lib/kubelet/pods/9ba52ba8-24c1-4b0c-83cb-6837e2353fa8/volumes" Oct 11 11:12:33.052240 master-1 kubenswrapper[4771]: I1011 11:12:33.052162 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/octavia-persistence-db-create-8rff9"] Oct 11 11:12:33.064936 master-1 kubenswrapper[4771]: I1011 11:12:33.064828 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/octavia-persistence-db-create-8rff9"] Oct 11 11:12:34.458674 master-1 kubenswrapper[4771]: I1011 11:12:34.458573 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f91fc642-f994-42aa-9bb1-589b5bda7c22" path="/var/lib/kubelet/pods/f91fc642-f994-42aa-9bb1-589b5bda7c22/volumes" Oct 11 11:12:45.068298 master-1 kubenswrapper[4771]: I1011 11:12:45.068179 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/octavia-5f0c-account-create-9njpc"] Oct 11 11:12:45.074451 master-1 kubenswrapper[4771]: I1011 11:12:45.073412 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/octavia-5f0c-account-create-9njpc"] Oct 11 11:12:46.450474 master-1 kubenswrapper[4771]: I1011 11:12:46.450391 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb36428-2086-42ca-8ebf-9864a0917971" path="/var/lib/kubelet/pods/3eb36428-2086-42ca-8ebf-9864a0917971/volumes" Oct 11 11:12:59.537402 master-1 kubenswrapper[4771]: I1011 11:12:59.537295 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/configure-os-dataplane-edpm-t4d7h"] Oct 11 11:12:59.540025 master-1 kubenswrapper[4771]: I1011 11:12:59.539973 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.544274 master-1 kubenswrapper[4771]: I1011 11:12:59.544221 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:12:59.553207 master-1 kubenswrapper[4771]: I1011 11:12:59.553128 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/configure-os-dataplane-edpm-t4d7h"] Oct 11 11:12:59.633688 master-1 kubenswrapper[4771]: I1011 11:12:59.633610 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-ssh-key\") pod \"configure-os-dataplane-edpm-t4d7h\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.633938 master-1 kubenswrapper[4771]: I1011 11:12:59.633698 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcl9\" (UniqueName: \"kubernetes.io/projected/89bef19a-554b-423c-b350-11b000bff73d-kube-api-access-hmcl9\") pod \"configure-os-dataplane-edpm-t4d7h\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.633984 master-1 kubenswrapper[4771]: I1011 11:12:59.633919 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-inventory\") pod \"configure-os-dataplane-edpm-t4d7h\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.735807 master-1 kubenswrapper[4771]: I1011 11:12:59.735732 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-ssh-key\") pod \"configure-os-dataplane-edpm-t4d7h\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.735807 master-1 kubenswrapper[4771]: I1011 11:12:59.735818 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcl9\" (UniqueName: \"kubernetes.io/projected/89bef19a-554b-423c-b350-11b000bff73d-kube-api-access-hmcl9\") pod \"configure-os-dataplane-edpm-t4d7h\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.736268 master-1 kubenswrapper[4771]: I1011 11:12:59.735939 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-inventory\") pod \"configure-os-dataplane-edpm-t4d7h\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.740426 master-1 kubenswrapper[4771]: I1011 11:12:59.740332 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-inventory\") pod \"configure-os-dataplane-edpm-t4d7h\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.742753 master-1 kubenswrapper[4771]: I1011 11:12:59.742641 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-ssh-key\") pod \"configure-os-dataplane-edpm-t4d7h\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.761724 master-1 kubenswrapper[4771]: I1011 11:12:59.761652 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcl9\" (UniqueName: \"kubernetes.io/projected/89bef19a-554b-423c-b350-11b000bff73d-kube-api-access-hmcl9\") pod \"configure-os-dataplane-edpm-t4d7h\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:12:59.868238 master-1 kubenswrapper[4771]: I1011 11:12:59.868133 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:13:00.484626 master-1 kubenswrapper[4771]: I1011 11:13:00.484100 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/configure-os-dataplane-edpm-t4d7h"] Oct 11 11:13:00.996113 master-1 kubenswrapper[4771]: I1011 11:13:00.996038 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-os-dataplane-edpm-t4d7h" event={"ID":"89bef19a-554b-423c-b350-11b000bff73d","Type":"ContainerStarted","Data":"10b24d028221fc67acf0e5a32b8e9a6276196774b4fb61c0a673ef5c09860f57"} Oct 11 11:13:02.016771 master-1 kubenswrapper[4771]: I1011 11:13:02.016679 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-os-dataplane-edpm-t4d7h" event={"ID":"89bef19a-554b-423c-b350-11b000bff73d","Type":"ContainerStarted","Data":"70de448794c96c4f984e2d3d72ad94feed3c008eba01f0c4522911ade0c71763"} Oct 11 11:13:02.044604 master-1 kubenswrapper[4771]: I1011 11:13:02.044471 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/configure-os-dataplane-edpm-t4d7h" podStartSLOduration=2.561380873 podStartE2EDuration="3.044441006s" podCreationTimestamp="2025-10-11 11:12:59 +0000 UTC" firstStartedPulling="2025-10-11 11:13:00.486159198 +0000 UTC m=+2812.460385649" lastFinishedPulling="2025-10-11 11:13:00.969219351 +0000 UTC m=+2812.943445782" observedRunningTime="2025-10-11 11:13:02.043421447 +0000 UTC m=+2814.017647928" watchObservedRunningTime="2025-10-11 11:13:02.044441006 +0000 UTC m=+2814.018667487" Oct 11 11:13:09.083183 master-1 kubenswrapper[4771]: I1011 11:13:09.083106 4771 generic.go:334] "Generic (PLEG): container finished" podID="24d0c33f-c29a-4878-a73d-4fdc54ee4754" containerID="8dfae439c8c304c6fb6f9f04b4870358e01be13642192b2cd7e8e013d0312a0b" exitCode=0 Oct 11 11:13:09.083183 master-1 kubenswrapper[4771]: I1011 11:13:09.083189 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/install-os-networker-deploy-networkers-6txzc" event={"ID":"24d0c33f-c29a-4878-a73d-4fdc54ee4754","Type":"ContainerDied","Data":"8dfae439c8c304c6fb6f9f04b4870358e01be13642192b2cd7e8e013d0312a0b"} Oct 11 11:13:10.805789 master-1 kubenswrapper[4771]: I1011 11:13:10.805711 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:13:10.900587 master-1 kubenswrapper[4771]: I1011 11:13:10.900452 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flrrm\" (UniqueName: \"kubernetes.io/projected/24d0c33f-c29a-4878-a73d-4fdc54ee4754-kube-api-access-flrrm\") pod \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " Oct 11 11:13:10.900922 master-1 kubenswrapper[4771]: I1011 11:13:10.900634 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-inventory\") pod \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " Oct 11 11:13:10.900922 master-1 kubenswrapper[4771]: I1011 11:13:10.900769 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-ssh-key\") pod \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\" (UID: \"24d0c33f-c29a-4878-a73d-4fdc54ee4754\") " Oct 11 11:13:10.906148 master-1 kubenswrapper[4771]: I1011 11:13:10.906036 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d0c33f-c29a-4878-a73d-4fdc54ee4754-kube-api-access-flrrm" (OuterVolumeSpecName: "kube-api-access-flrrm") pod "24d0c33f-c29a-4878-a73d-4fdc54ee4754" (UID: "24d0c33f-c29a-4878-a73d-4fdc54ee4754"). InnerVolumeSpecName "kube-api-access-flrrm". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:13:10.939432 master-1 kubenswrapper[4771]: I1011 11:13:10.939322 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "24d0c33f-c29a-4878-a73d-4fdc54ee4754" (UID: "24d0c33f-c29a-4878-a73d-4fdc54ee4754"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:13:10.950020 master-1 kubenswrapper[4771]: I1011 11:13:10.948642 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-inventory" (OuterVolumeSpecName: "inventory") pod "24d0c33f-c29a-4878-a73d-4fdc54ee4754" (UID: "24d0c33f-c29a-4878-a73d-4fdc54ee4754"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:13:11.003972 master-1 kubenswrapper[4771]: I1011 11:13:11.003885 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-flrrm\" (UniqueName: \"kubernetes.io/projected/24d0c33f-c29a-4878-a73d-4fdc54ee4754-kube-api-access-flrrm\") on node \"master-1\" DevicePath \"\"" Oct 11 11:13:11.003972 master-1 kubenswrapper[4771]: I1011 11:13:11.003941 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:13:11.003972 master-1 kubenswrapper[4771]: I1011 11:13:11.003952 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/24d0c33f-c29a-4878-a73d-4fdc54ee4754-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:13:11.109845 master-1 kubenswrapper[4771]: I1011 11:13:11.109749 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/install-os-networker-deploy-networkers-6txzc" event={"ID":"24d0c33f-c29a-4878-a73d-4fdc54ee4754","Type":"ContainerDied","Data":"358ed73fdc20984b748856c249dfddc8dcd94200b65b4cf2e53a249cbb6512a0"} Oct 11 11:13:11.109845 master-1 kubenswrapper[4771]: I1011 11:13:11.109811 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="358ed73fdc20984b748856c249dfddc8dcd94200b65b4cf2e53a249cbb6512a0" Oct 11 11:13:11.110275 master-1 kubenswrapper[4771]: I1011 11:13:11.109906 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/install-os-networker-deploy-networkers-6txzc" Oct 11 11:13:21.073582 master-1 kubenswrapper[4771]: I1011 11:13:21.073475 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openstack/octavia-db-sync-dsphh"] Oct 11 11:13:21.085317 master-1 kubenswrapper[4771]: I1011 11:13:21.085212 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openstack/octavia-db-sync-dsphh"] Oct 11 11:13:22.451706 master-1 kubenswrapper[4771]: I1011 11:13:22.451624 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2681ddac-5e31-449b-bf71-fb54e8ba389c" path="/var/lib/kubelet/pods/2681ddac-5e31-449b-bf71-fb54e8ba389c/volumes" Oct 11 11:13:24.953989 master-1 kubenswrapper[4771]: I1011 11:13:24.953874 4771 scope.go:117] "RemoveContainer" containerID="649f9407421e9587f48e63d8c49281e9025d233acd0bf83983ed52b03d671758" Oct 11 11:13:24.988450 master-1 kubenswrapper[4771]: I1011 11:13:24.988342 4771 scope.go:117] "RemoveContainer" containerID="9343b079fc4a104c0cb1564885c29bbb8cf2e14829a4f1096f11a9696ef57edf" Oct 11 11:13:25.073109 master-1 kubenswrapper[4771]: I1011 11:13:25.072953 4771 scope.go:117] "RemoveContainer" containerID="24ea528e7f6dc70693d8dee3aad4fc9efa2cfed93954344bd9c5720391f051ef" Oct 11 11:13:25.139564 master-1 kubenswrapper[4771]: I1011 11:13:25.139488 4771 scope.go:117] "RemoveContainer" containerID="00d99e2ba51100c415ae5d1a3b19ce4ab68cd0b4655796bec9fd8f7ec75f10f8" Oct 11 11:13:25.198574 master-1 kubenswrapper[4771]: I1011 11:13:25.198507 4771 scope.go:117] "RemoveContainer" containerID="2433d4bbed13285c1fb5cb4b22ca8e93fb7e88d52d9a41f34c5f718dbcf8c96b" Oct 11 11:13:55.634389 master-1 kubenswrapper[4771]: I1011 11:13:55.634264 4771 generic.go:334] "Generic (PLEG): container finished" podID="89bef19a-554b-423c-b350-11b000bff73d" containerID="70de448794c96c4f984e2d3d72ad94feed3c008eba01f0c4522911ade0c71763" exitCode=2 Oct 11 11:13:55.635428 master-1 kubenswrapper[4771]: I1011 11:13:55.634381 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-os-dataplane-edpm-t4d7h" event={"ID":"89bef19a-554b-423c-b350-11b000bff73d","Type":"ContainerDied","Data":"70de448794c96c4f984e2d3d72ad94feed3c008eba01f0c4522911ade0c71763"} Oct 11 11:13:57.285195 master-1 kubenswrapper[4771]: I1011 11:13:57.285106 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:13:57.483736 master-1 kubenswrapper[4771]: I1011 11:13:57.483582 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-inventory\") pod \"89bef19a-554b-423c-b350-11b000bff73d\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " Oct 11 11:13:57.484090 master-1 kubenswrapper[4771]: I1011 11:13:57.483788 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmcl9\" (UniqueName: \"kubernetes.io/projected/89bef19a-554b-423c-b350-11b000bff73d-kube-api-access-hmcl9\") pod \"89bef19a-554b-423c-b350-11b000bff73d\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " Oct 11 11:13:57.484090 master-1 kubenswrapper[4771]: I1011 11:13:57.483871 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-ssh-key\") pod \"89bef19a-554b-423c-b350-11b000bff73d\" (UID: \"89bef19a-554b-423c-b350-11b000bff73d\") " Oct 11 11:13:57.489440 master-1 kubenswrapper[4771]: I1011 11:13:57.489332 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bef19a-554b-423c-b350-11b000bff73d-kube-api-access-hmcl9" (OuterVolumeSpecName: "kube-api-access-hmcl9") pod "89bef19a-554b-423c-b350-11b000bff73d" (UID: "89bef19a-554b-423c-b350-11b000bff73d"). InnerVolumeSpecName "kube-api-access-hmcl9". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:13:57.513722 master-1 kubenswrapper[4771]: I1011 11:13:57.513621 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-inventory" (OuterVolumeSpecName: "inventory") pod "89bef19a-554b-423c-b350-11b000bff73d" (UID: "89bef19a-554b-423c-b350-11b000bff73d"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:13:57.514582 master-1 kubenswrapper[4771]: I1011 11:13:57.514510 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "89bef19a-554b-423c-b350-11b000bff73d" (UID: "89bef19a-554b-423c-b350-11b000bff73d"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:13:57.586640 master-1 kubenswrapper[4771]: I1011 11:13:57.586529 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:13:57.586640 master-1 kubenswrapper[4771]: I1011 11:13:57.586577 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-hmcl9\" (UniqueName: \"kubernetes.io/projected/89bef19a-554b-423c-b350-11b000bff73d-kube-api-access-hmcl9\") on node \"master-1\" DevicePath \"\"" Oct 11 11:13:57.586640 master-1 kubenswrapper[4771]: I1011 11:13:57.586604 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/89bef19a-554b-423c-b350-11b000bff73d-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:13:57.656859 master-1 kubenswrapper[4771]: I1011 11:13:57.656737 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-os-dataplane-edpm-t4d7h" event={"ID":"89bef19a-554b-423c-b350-11b000bff73d","Type":"ContainerDied","Data":"10b24d028221fc67acf0e5a32b8e9a6276196774b4fb61c0a673ef5c09860f57"} Oct 11 11:13:57.656859 master-1 kubenswrapper[4771]: I1011 11:13:57.656815 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10b24d028221fc67acf0e5a32b8e9a6276196774b4fb61c0a673ef5c09860f57" Oct 11 11:13:57.657475 master-1 kubenswrapper[4771]: I1011 11:13:57.656944 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/configure-os-dataplane-edpm-t4d7h" Oct 11 11:14:05.475117 master-1 kubenswrapper[4771]: I1011 11:14:05.474986 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/configure-os-dataplane-edpm-kndnx"] Oct 11 11:14:05.476274 master-1 kubenswrapper[4771]: E1011 11:14:05.475626 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="24d0c33f-c29a-4878-a73d-4fdc54ee4754" containerName="install-os-networker-deploy-networkers" Oct 11 11:14:05.476274 master-1 kubenswrapper[4771]: I1011 11:14:05.475715 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d0c33f-c29a-4878-a73d-4fdc54ee4754" containerName="install-os-networker-deploy-networkers" Oct 11 11:14:05.476274 master-1 kubenswrapper[4771]: E1011 11:14:05.475792 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="89bef19a-554b-423c-b350-11b000bff73d" containerName="configure-os-dataplane-edpm" Oct 11 11:14:05.476274 master-1 kubenswrapper[4771]: I1011 11:14:05.475801 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bef19a-554b-423c-b350-11b000bff73d" containerName="configure-os-dataplane-edpm" Oct 11 11:14:05.476642 master-1 kubenswrapper[4771]: I1011 11:14:05.476304 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="24d0c33f-c29a-4878-a73d-4fdc54ee4754" containerName="install-os-networker-deploy-networkers" Oct 11 11:14:05.476642 master-1 kubenswrapper[4771]: I1011 11:14:05.476325 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="89bef19a-554b-423c-b350-11b000bff73d" containerName="configure-os-dataplane-edpm" Oct 11 11:14:05.477518 master-1 kubenswrapper[4771]: I1011 11:14:05.477469 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.480684 master-1 kubenswrapper[4771]: I1011 11:14:05.480605 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:14:05.481124 master-1 kubenswrapper[4771]: I1011 11:14:05.481056 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:14:05.481221 master-1 kubenswrapper[4771]: I1011 11:14:05.481073 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:14:05.489572 master-1 kubenswrapper[4771]: I1011 11:14:05.489487 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/configure-os-dataplane-edpm-kndnx"] Oct 11 11:14:05.492397 master-1 kubenswrapper[4771]: I1011 11:14:05.492301 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-ssh-key\") pod \"configure-os-dataplane-edpm-kndnx\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.492671 master-1 kubenswrapper[4771]: I1011 11:14:05.492626 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-inventory\") pod \"configure-os-dataplane-edpm-kndnx\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.492763 master-1 kubenswrapper[4771]: I1011 11:14:05.492671 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5dv8\" (UniqueName: \"kubernetes.io/projected/453fb91b-f029-45ef-abab-ee2e81fe09d4-kube-api-access-j5dv8\") pod \"configure-os-dataplane-edpm-kndnx\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.595875 master-1 kubenswrapper[4771]: I1011 11:14:05.595735 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-inventory\") pod \"configure-os-dataplane-edpm-kndnx\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.595875 master-1 kubenswrapper[4771]: I1011 11:14:05.595850 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-j5dv8\" (UniqueName: \"kubernetes.io/projected/453fb91b-f029-45ef-abab-ee2e81fe09d4-kube-api-access-j5dv8\") pod \"configure-os-dataplane-edpm-kndnx\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.596320 master-1 kubenswrapper[4771]: I1011 11:14:05.595984 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-ssh-key\") pod \"configure-os-dataplane-edpm-kndnx\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.602458 master-1 kubenswrapper[4771]: I1011 11:14:05.602399 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-inventory\") pod \"configure-os-dataplane-edpm-kndnx\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.602634 master-1 kubenswrapper[4771]: I1011 11:14:05.602588 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-ssh-key\") pod \"configure-os-dataplane-edpm-kndnx\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.617172 master-1 kubenswrapper[4771]: I1011 11:14:05.617101 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5dv8\" (UniqueName: \"kubernetes.io/projected/453fb91b-f029-45ef-abab-ee2e81fe09d4-kube-api-access-j5dv8\") pod \"configure-os-dataplane-edpm-kndnx\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:05.817893 master-1 kubenswrapper[4771]: I1011 11:14:05.817792 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:06.496931 master-1 kubenswrapper[4771]: I1011 11:14:06.496850 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/configure-os-dataplane-edpm-kndnx"] Oct 11 11:14:06.818907 master-1 kubenswrapper[4771]: I1011 11:14:06.818811 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-os-dataplane-edpm-kndnx" event={"ID":"453fb91b-f029-45ef-abab-ee2e81fe09d4","Type":"ContainerStarted","Data":"9b21e0d2161f7c72225ba0d8f1f34b86ffb2212388c14f76d64766036b46fcbe"} Oct 11 11:14:07.829289 master-1 kubenswrapper[4771]: I1011 11:14:07.829175 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-os-dataplane-edpm-kndnx" event={"ID":"453fb91b-f029-45ef-abab-ee2e81fe09d4","Type":"ContainerStarted","Data":"e3721d28b852247005699682d334c78c4d7fde3a9e258d447e6fee94baa45c00"} Oct 11 11:14:07.860897 master-1 kubenswrapper[4771]: I1011 11:14:07.860782 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/configure-os-dataplane-edpm-kndnx" podStartSLOduration=2.183146566 podStartE2EDuration="2.86073012s" podCreationTimestamp="2025-10-11 11:14:05 +0000 UTC" firstStartedPulling="2025-10-11 11:14:06.531541904 +0000 UTC m=+2878.505768355" lastFinishedPulling="2025-10-11 11:14:07.209125428 +0000 UTC m=+2879.183351909" observedRunningTime="2025-10-11 11:14:07.855318646 +0000 UTC m=+2879.829545087" watchObservedRunningTime="2025-10-11 11:14:07.86073012 +0000 UTC m=+2879.834956551" Oct 11 11:14:55.333038 master-1 kubenswrapper[4771]: I1011 11:14:55.332955 4771 generic.go:334] "Generic (PLEG): container finished" podID="453fb91b-f029-45ef-abab-ee2e81fe09d4" containerID="e3721d28b852247005699682d334c78c4d7fde3a9e258d447e6fee94baa45c00" exitCode=0 Oct 11 11:14:55.333038 master-1 kubenswrapper[4771]: I1011 11:14:55.333028 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-os-dataplane-edpm-kndnx" event={"ID":"453fb91b-f029-45ef-abab-ee2e81fe09d4","Type":"ContainerDied","Data":"e3721d28b852247005699682d334c78c4d7fde3a9e258d447e6fee94baa45c00"} Oct 11 11:14:56.981044 master-1 kubenswrapper[4771]: I1011 11:14:56.980962 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:57.090810 master-1 kubenswrapper[4771]: I1011 11:14:57.090642 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5dv8\" (UniqueName: \"kubernetes.io/projected/453fb91b-f029-45ef-abab-ee2e81fe09d4-kube-api-access-j5dv8\") pod \"453fb91b-f029-45ef-abab-ee2e81fe09d4\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " Oct 11 11:14:57.091224 master-1 kubenswrapper[4771]: I1011 11:14:57.090933 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-ssh-key\") pod \"453fb91b-f029-45ef-abab-ee2e81fe09d4\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " Oct 11 11:14:57.091224 master-1 kubenswrapper[4771]: I1011 11:14:57.091143 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-inventory\") pod \"453fb91b-f029-45ef-abab-ee2e81fe09d4\" (UID: \"453fb91b-f029-45ef-abab-ee2e81fe09d4\") " Oct 11 11:14:57.097492 master-1 kubenswrapper[4771]: I1011 11:14:57.094383 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453fb91b-f029-45ef-abab-ee2e81fe09d4-kube-api-access-j5dv8" (OuterVolumeSpecName: "kube-api-access-j5dv8") pod "453fb91b-f029-45ef-abab-ee2e81fe09d4" (UID: "453fb91b-f029-45ef-abab-ee2e81fe09d4"). InnerVolumeSpecName "kube-api-access-j5dv8". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:14:57.113658 master-1 kubenswrapper[4771]: I1011 11:14:57.113488 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-inventory" (OuterVolumeSpecName: "inventory") pod "453fb91b-f029-45ef-abab-ee2e81fe09d4" (UID: "453fb91b-f029-45ef-abab-ee2e81fe09d4"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:14:57.122094 master-1 kubenswrapper[4771]: I1011 11:14:57.122040 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "453fb91b-f029-45ef-abab-ee2e81fe09d4" (UID: "453fb91b-f029-45ef-abab-ee2e81fe09d4"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:14:57.194606 master-1 kubenswrapper[4771]: I1011 11:14:57.194535 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:14:57.194606 master-1 kubenswrapper[4771]: I1011 11:14:57.194590 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-j5dv8\" (UniqueName: \"kubernetes.io/projected/453fb91b-f029-45ef-abab-ee2e81fe09d4-kube-api-access-j5dv8\") on node \"master-1\" DevicePath \"\"" Oct 11 11:14:57.194606 master-1 kubenswrapper[4771]: I1011 11:14:57.194606 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/453fb91b-f029-45ef-abab-ee2e81fe09d4-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:14:57.353733 master-1 kubenswrapper[4771]: I1011 11:14:57.353641 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/configure-os-dataplane-edpm-kndnx" event={"ID":"453fb91b-f029-45ef-abab-ee2e81fe09d4","Type":"ContainerDied","Data":"9b21e0d2161f7c72225ba0d8f1f34b86ffb2212388c14f76d64766036b46fcbe"} Oct 11 11:14:57.353733 master-1 kubenswrapper[4771]: I1011 11:14:57.353710 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b21e0d2161f7c72225ba0d8f1f34b86ffb2212388c14f76d64766036b46fcbe" Oct 11 11:14:57.353733 master-1 kubenswrapper[4771]: I1011 11:14:57.353721 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/configure-os-dataplane-edpm-kndnx" Oct 11 11:14:57.505542 master-1 kubenswrapper[4771]: I1011 11:14:57.505462 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ssh-known-hosts-dataplane-wv265"] Oct 11 11:14:57.505913 master-1 kubenswrapper[4771]: E1011 11:14:57.505898 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="453fb91b-f029-45ef-abab-ee2e81fe09d4" containerName="configure-os-dataplane-edpm" Oct 11 11:14:57.505994 master-1 kubenswrapper[4771]: I1011 11:14:57.505914 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="453fb91b-f029-45ef-abab-ee2e81fe09d4" containerName="configure-os-dataplane-edpm" Oct 11 11:14:57.506771 master-1 kubenswrapper[4771]: I1011 11:14:57.506739 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="453fb91b-f029-45ef-abab-ee2e81fe09d4" containerName="configure-os-dataplane-edpm" Oct 11 11:14:57.508240 master-1 kubenswrapper[4771]: I1011 11:14:57.508203 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.512679 master-1 kubenswrapper[4771]: I1011 11:14:57.512630 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:14:57.512989 master-1 kubenswrapper[4771]: I1011 11:14:57.512648 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:14:57.513163 master-1 kubenswrapper[4771]: I1011 11:14:57.513100 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:14:57.519811 master-1 kubenswrapper[4771]: I1011 11:14:57.519740 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ssh-known-hosts-dataplane-wv265"] Oct 11 11:14:57.606496 master-1 kubenswrapper[4771]: I1011 11:14:57.606422 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory-0\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-inventory-0\") pod \"ssh-known-hosts-dataplane-wv265\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.606496 master-1 kubenswrapper[4771]: I1011 11:14:57.606487 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59cj\" (UniqueName: \"kubernetes.io/projected/e0206b4d-5dcd-4fef-9265-f952c8f0023c-kube-api-access-p59cj\") pod \"ssh-known-hosts-dataplane-wv265\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.606942 master-1 kubenswrapper[4771]: I1011 11:14:57.606852 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key-edpm\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-ssh-key-edpm\") pod \"ssh-known-hosts-dataplane-wv265\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.710145 master-1 kubenswrapper[4771]: I1011 11:14:57.709972 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key-edpm\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-ssh-key-edpm\") pod \"ssh-known-hosts-dataplane-wv265\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.710473 master-1 kubenswrapper[4771]: I1011 11:14:57.710195 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory-0\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-inventory-0\") pod \"ssh-known-hosts-dataplane-wv265\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.710473 master-1 kubenswrapper[4771]: I1011 11:14:57.710237 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-p59cj\" (UniqueName: \"kubernetes.io/projected/e0206b4d-5dcd-4fef-9265-f952c8f0023c-kube-api-access-p59cj\") pod \"ssh-known-hosts-dataplane-wv265\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.714599 master-1 kubenswrapper[4771]: I1011 11:14:57.714529 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory-0\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-inventory-0\") pod \"ssh-known-hosts-dataplane-wv265\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.714675 master-1 kubenswrapper[4771]: I1011 11:14:57.714624 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key-edpm\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-ssh-key-edpm\") pod \"ssh-known-hosts-dataplane-wv265\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.734524 master-1 kubenswrapper[4771]: I1011 11:14:57.734465 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59cj\" (UniqueName: \"kubernetes.io/projected/e0206b4d-5dcd-4fef-9265-f952c8f0023c-kube-api-access-p59cj\") pod \"ssh-known-hosts-dataplane-wv265\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:57.836852 master-1 kubenswrapper[4771]: I1011 11:14:57.836757 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:14:58.417536 master-1 kubenswrapper[4771]: I1011 11:14:58.417452 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ssh-known-hosts-dataplane-wv265"] Oct 11 11:14:59.380460 master-1 kubenswrapper[4771]: I1011 11:14:59.380327 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ssh-known-hosts-dataplane-wv265" event={"ID":"e0206b4d-5dcd-4fef-9265-f952c8f0023c","Type":"ContainerStarted","Data":"ad9ad2f14084bffb4b6b64c241efd045693af3e205d809c8184e6e7d059cbc2c"} Oct 11 11:14:59.380460 master-1 kubenswrapper[4771]: I1011 11:14:59.380452 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ssh-known-hosts-dataplane-wv265" event={"ID":"e0206b4d-5dcd-4fef-9265-f952c8f0023c","Type":"ContainerStarted","Data":"2da56407dab575ae818c42606fee39da936da62ad1fe6ce7dc8cd66a01e86623"} Oct 11 11:14:59.413574 master-1 kubenswrapper[4771]: I1011 11:14:59.413172 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ssh-known-hosts-dataplane-wv265" podStartSLOduration=1.987718166 podStartE2EDuration="2.413146331s" podCreationTimestamp="2025-10-11 11:14:57 +0000 UTC" firstStartedPulling="2025-10-11 11:14:58.420226509 +0000 UTC m=+2930.394452950" lastFinishedPulling="2025-10-11 11:14:58.845654654 +0000 UTC m=+2930.819881115" observedRunningTime="2025-10-11 11:14:59.407369236 +0000 UTC m=+2931.381595677" watchObservedRunningTime="2025-10-11 11:14:59.413146331 +0000 UTC m=+2931.387372772" Oct 11 11:15:00.179730 master-1 kubenswrapper[4771]: I1011 11:15:00.179635 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg"] Oct 11 11:15:00.181941 master-1 kubenswrapper[4771]: I1011 11:15:00.181895 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.185781 master-1 kubenswrapper[4771]: I1011 11:15:00.185708 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-operator-lifecycle-manager"/"collect-profiles-dockercfg-hbjq2" Oct 11 11:15:00.186288 master-1 kubenswrapper[4771]: I1011 11:15:00.186198 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-operator-lifecycle-manager"/"pprof-cert" Oct 11 11:15:00.187551 master-1 kubenswrapper[4771]: I1011 11:15:00.187512 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-operator-lifecycle-manager"/"collect-profiles-config" Oct 11 11:15:00.189296 master-1 kubenswrapper[4771]: I1011 11:15:00.189217 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg"] Oct 11 11:15:00.375160 master-1 kubenswrapper[4771]: I1011 11:15:00.375034 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmrz6\" (UniqueName: \"kubernetes.io/projected/9e6bbddb-86b5-4be5-b585-b318e8254f1e-kube-api-access-rmrz6\") pod \"collect-profiles-29336355-mmqkg\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.376166 master-1 kubenswrapper[4771]: I1011 11:15:00.376071 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/9e6bbddb-86b5-4be5-b585-b318e8254f1e-secret-volume\") pod \"collect-profiles-29336355-mmqkg\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.376610 master-1 kubenswrapper[4771]: I1011 11:15:00.376568 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6bbddb-86b5-4be5-b585-b318e8254f1e-config-volume\") pod \"collect-profiles-29336355-mmqkg\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.480677 master-1 kubenswrapper[4771]: I1011 11:15:00.480467 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/9e6bbddb-86b5-4be5-b585-b318e8254f1e-secret-volume\") pod \"collect-profiles-29336355-mmqkg\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.481016 master-1 kubenswrapper[4771]: I1011 11:15:00.480965 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6bbddb-86b5-4be5-b585-b318e8254f1e-config-volume\") pod \"collect-profiles-29336355-mmqkg\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.481195 master-1 kubenswrapper[4771]: I1011 11:15:00.481147 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-rmrz6\" (UniqueName: \"kubernetes.io/projected/9e6bbddb-86b5-4be5-b585-b318e8254f1e-kube-api-access-rmrz6\") pod \"collect-profiles-29336355-mmqkg\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.482142 master-1 kubenswrapper[4771]: I1011 11:15:00.482084 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6bbddb-86b5-4be5-b585-b318e8254f1e-config-volume\") pod \"collect-profiles-29336355-mmqkg\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.516627 master-1 kubenswrapper[4771]: I1011 11:15:00.516551 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/9e6bbddb-86b5-4be5-b585-b318e8254f1e-secret-volume\") pod \"collect-profiles-29336355-mmqkg\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.521686 master-1 kubenswrapper[4771]: I1011 11:15:00.520119 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmrz6\" (UniqueName: \"kubernetes.io/projected/9e6bbddb-86b5-4be5-b585-b318e8254f1e-kube-api-access-rmrz6\") pod \"collect-profiles-29336355-mmqkg\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:00.816268 master-1 kubenswrapper[4771]: I1011 11:15:00.816152 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:01.320490 master-1 kubenswrapper[4771]: I1011 11:15:01.316690 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg"] Oct 11 11:15:01.405501 master-1 kubenswrapper[4771]: I1011 11:15:01.405421 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" event={"ID":"9e6bbddb-86b5-4be5-b585-b318e8254f1e","Type":"ContainerStarted","Data":"f6867e711943ee7453d11ebd357f7902a3afb229518b252be5c7f06ed5a4e6ab"} Oct 11 11:15:02.420269 master-1 kubenswrapper[4771]: I1011 11:15:02.420183 4771 generic.go:334] "Generic (PLEG): container finished" podID="9e6bbddb-86b5-4be5-b585-b318e8254f1e" containerID="8ab819b3c7d07574afc9eec1d49bf5d6d0fcbbd18337a802948a48143d1a402e" exitCode=0 Oct 11 11:15:02.420269 master-1 kubenswrapper[4771]: I1011 11:15:02.420237 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" event={"ID":"9e6bbddb-86b5-4be5-b585-b318e8254f1e","Type":"ContainerDied","Data":"8ab819b3c7d07574afc9eec1d49bf5d6d0fcbbd18337a802948a48143d1a402e"} Oct 11 11:15:03.962828 master-1 kubenswrapper[4771]: I1011 11:15:03.962736 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:04.084856 master-1 kubenswrapper[4771]: I1011 11:15:04.084786 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/9e6bbddb-86b5-4be5-b585-b318e8254f1e-secret-volume\") pod \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " Oct 11 11:15:04.085080 master-1 kubenswrapper[4771]: I1011 11:15:04.085042 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmrz6\" (UniqueName: \"kubernetes.io/projected/9e6bbddb-86b5-4be5-b585-b318e8254f1e-kube-api-access-rmrz6\") pod \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " Oct 11 11:15:04.086029 master-1 kubenswrapper[4771]: I1011 11:15:04.085985 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6bbddb-86b5-4be5-b585-b318e8254f1e-config-volume" (OuterVolumeSpecName: "config-volume") pod "9e6bbddb-86b5-4be5-b585-b318e8254f1e" (UID: "9e6bbddb-86b5-4be5-b585-b318e8254f1e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:15:04.086103 master-1 kubenswrapper[4771]: I1011 11:15:04.085351 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6bbddb-86b5-4be5-b585-b318e8254f1e-config-volume\") pod \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\" (UID: \"9e6bbddb-86b5-4be5-b585-b318e8254f1e\") " Oct 11 11:15:04.086875 master-1 kubenswrapper[4771]: I1011 11:15:04.086828 4771 reconciler_common.go:293] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6bbddb-86b5-4be5-b585-b318e8254f1e-config-volume\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:04.090599 master-1 kubenswrapper[4771]: I1011 11:15:04.090555 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6bbddb-86b5-4be5-b585-b318e8254f1e-secret-volume" (OuterVolumeSpecName: "secret-volume") pod "9e6bbddb-86b5-4be5-b585-b318e8254f1e" (UID: "9e6bbddb-86b5-4be5-b585-b318e8254f1e"). InnerVolumeSpecName "secret-volume". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:15:04.094316 master-1 kubenswrapper[4771]: I1011 11:15:04.094245 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6bbddb-86b5-4be5-b585-b318e8254f1e-kube-api-access-rmrz6" (OuterVolumeSpecName: "kube-api-access-rmrz6") pod "9e6bbddb-86b5-4be5-b585-b318e8254f1e" (UID: "9e6bbddb-86b5-4be5-b585-b318e8254f1e"). InnerVolumeSpecName "kube-api-access-rmrz6". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:15:04.189908 master-1 kubenswrapper[4771]: I1011 11:15:04.189838 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-rmrz6\" (UniqueName: \"kubernetes.io/projected/9e6bbddb-86b5-4be5-b585-b318e8254f1e-kube-api-access-rmrz6\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:04.189908 master-1 kubenswrapper[4771]: I1011 11:15:04.189897 4771 reconciler_common.go:293] "Volume detached for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/9e6bbddb-86b5-4be5-b585-b318e8254f1e-secret-volume\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:04.445932 master-1 kubenswrapper[4771]: I1011 11:15:04.445765 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" Oct 11 11:15:04.454521 master-1 kubenswrapper[4771]: I1011 11:15:04.454432 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336355-mmqkg" event={"ID":"9e6bbddb-86b5-4be5-b585-b318e8254f1e","Type":"ContainerDied","Data":"f6867e711943ee7453d11ebd357f7902a3afb229518b252be5c7f06ed5a4e6ab"} Oct 11 11:15:04.454521 master-1 kubenswrapper[4771]: I1011 11:15:04.454504 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6867e711943ee7453d11ebd357f7902a3afb229518b252be5c7f06ed5a4e6ab" Oct 11 11:15:05.094373 master-1 kubenswrapper[4771]: I1011 11:15:05.094281 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v"] Oct 11 11:15:05.102391 master-1 kubenswrapper[4771]: I1011 11:15:05.102306 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336310-8nc4v"] Oct 11 11:15:06.454153 master-1 kubenswrapper[4771]: I1011 11:15:06.454050 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4904d67-3c44-40d9-8ea8-026d727e9486" path="/var/lib/kubelet/pods/f4904d67-3c44-40d9-8ea8-026d727e9486/volumes" Oct 11 11:15:06.470327 master-1 kubenswrapper[4771]: I1011 11:15:06.470053 4771 generic.go:334] "Generic (PLEG): container finished" podID="e0206b4d-5dcd-4fef-9265-f952c8f0023c" containerID="ad9ad2f14084bffb4b6b64c241efd045693af3e205d809c8184e6e7d059cbc2c" exitCode=0 Oct 11 11:15:06.470327 master-1 kubenswrapper[4771]: I1011 11:15:06.470132 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ssh-known-hosts-dataplane-wv265" event={"ID":"e0206b4d-5dcd-4fef-9265-f952c8f0023c","Type":"ContainerDied","Data":"ad9ad2f14084bffb4b6b64c241efd045693af3e205d809c8184e6e7d059cbc2c"} Oct 11 11:15:08.144561 master-1 kubenswrapper[4771]: I1011 11:15:08.144513 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:15:08.189910 master-1 kubenswrapper[4771]: I1011 11:15:08.189829 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory-0\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-inventory-0\") pod \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " Oct 11 11:15:08.190150 master-1 kubenswrapper[4771]: I1011 11:15:08.190048 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p59cj\" (UniqueName: \"kubernetes.io/projected/e0206b4d-5dcd-4fef-9265-f952c8f0023c-kube-api-access-p59cj\") pod \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " Oct 11 11:15:08.190443 master-1 kubenswrapper[4771]: I1011 11:15:08.190408 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key-edpm\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-ssh-key-edpm\") pod \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\" (UID: \"e0206b4d-5dcd-4fef-9265-f952c8f0023c\") " Oct 11 11:15:08.196934 master-1 kubenswrapper[4771]: I1011 11:15:08.196853 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0206b4d-5dcd-4fef-9265-f952c8f0023c-kube-api-access-p59cj" (OuterVolumeSpecName: "kube-api-access-p59cj") pod "e0206b4d-5dcd-4fef-9265-f952c8f0023c" (UID: "e0206b4d-5dcd-4fef-9265-f952c8f0023c"). InnerVolumeSpecName "kube-api-access-p59cj". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:15:08.213570 master-1 kubenswrapper[4771]: I1011 11:15:08.213477 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-inventory-0" (OuterVolumeSpecName: "inventory-0") pod "e0206b4d-5dcd-4fef-9265-f952c8f0023c" (UID: "e0206b4d-5dcd-4fef-9265-f952c8f0023c"). InnerVolumeSpecName "inventory-0". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:15:08.216928 master-1 kubenswrapper[4771]: I1011 11:15:08.216850 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-ssh-key-edpm" (OuterVolumeSpecName: "ssh-key-edpm") pod "e0206b4d-5dcd-4fef-9265-f952c8f0023c" (UID: "e0206b4d-5dcd-4fef-9265-f952c8f0023c"). InnerVolumeSpecName "ssh-key-edpm". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:15:08.293722 master-1 kubenswrapper[4771]: I1011 11:15:08.293628 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key-edpm\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-ssh-key-edpm\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:08.293722 master-1 kubenswrapper[4771]: I1011 11:15:08.293686 4771 reconciler_common.go:293] "Volume detached for volume \"inventory-0\" (UniqueName: \"kubernetes.io/secret/e0206b4d-5dcd-4fef-9265-f952c8f0023c-inventory-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:08.293722 master-1 kubenswrapper[4771]: I1011 11:15:08.293699 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-p59cj\" (UniqueName: \"kubernetes.io/projected/e0206b4d-5dcd-4fef-9265-f952c8f0023c-kube-api-access-p59cj\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:08.491289 master-1 kubenswrapper[4771]: I1011 11:15:08.491228 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ssh-known-hosts-dataplane-wv265" event={"ID":"e0206b4d-5dcd-4fef-9265-f952c8f0023c","Type":"ContainerDied","Data":"2da56407dab575ae818c42606fee39da936da62ad1fe6ce7dc8cd66a01e86623"} Oct 11 11:15:08.491464 master-1 kubenswrapper[4771]: I1011 11:15:08.491292 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2da56407dab575ae818c42606fee39da936da62ad1fe6ce7dc8cd66a01e86623" Oct 11 11:15:08.491464 master-1 kubenswrapper[4771]: I1011 11:15:08.491389 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ssh-known-hosts-dataplane-wv265" Oct 11 11:15:08.788331 master-1 kubenswrapper[4771]: I1011 11:15:08.788069 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/run-os-dataplane-edpm-2b9k2"] Oct 11 11:15:08.789016 master-1 kubenswrapper[4771]: E1011 11:15:08.788955 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e0206b4d-5dcd-4fef-9265-f952c8f0023c" containerName="ssh-known-hosts-dataplane" Oct 11 11:15:08.789016 master-1 kubenswrapper[4771]: I1011 11:15:08.788997 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0206b4d-5dcd-4fef-9265-f952c8f0023c" containerName="ssh-known-hosts-dataplane" Oct 11 11:15:08.789211 master-1 kubenswrapper[4771]: E1011 11:15:08.789031 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="9e6bbddb-86b5-4be5-b585-b318e8254f1e" containerName="collect-profiles" Oct 11 11:15:08.789211 master-1 kubenswrapper[4771]: I1011 11:15:08.789046 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e6bbddb-86b5-4be5-b585-b318e8254f1e" containerName="collect-profiles" Oct 11 11:15:08.789428 master-1 kubenswrapper[4771]: I1011 11:15:08.789350 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e0206b4d-5dcd-4fef-9265-f952c8f0023c" containerName="ssh-known-hosts-dataplane" Oct 11 11:15:08.789596 master-1 kubenswrapper[4771]: I1011 11:15:08.789564 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="9e6bbddb-86b5-4be5-b585-b318e8254f1e" containerName="collect-profiles" Oct 11 11:15:08.790881 master-1 kubenswrapper[4771]: I1011 11:15:08.790828 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:08.795472 master-1 kubenswrapper[4771]: I1011 11:15:08.795334 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:15:08.795957 master-1 kubenswrapper[4771]: I1011 11:15:08.795876 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:15:08.795957 master-1 kubenswrapper[4771]: I1011 11:15:08.795887 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:15:08.804393 master-1 kubenswrapper[4771]: I1011 11:15:08.802556 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/run-os-dataplane-edpm-2b9k2"] Oct 11 11:15:08.817423 master-1 kubenswrapper[4771]: I1011 11:15:08.817289 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-ssh-key\") pod \"run-os-dataplane-edpm-2b9k2\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:08.817865 master-1 kubenswrapper[4771]: I1011 11:15:08.817616 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-inventory\") pod \"run-os-dataplane-edpm-2b9k2\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:08.817865 master-1 kubenswrapper[4771]: I1011 11:15:08.817681 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkqqv\" (UniqueName: \"kubernetes.io/projected/c7fb916b-469a-46ca-806b-bd61c8b674fd-kube-api-access-lkqqv\") pod \"run-os-dataplane-edpm-2b9k2\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:08.920498 master-1 kubenswrapper[4771]: I1011 11:15:08.920413 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-inventory\") pod \"run-os-dataplane-edpm-2b9k2\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:08.920867 master-1 kubenswrapper[4771]: I1011 11:15:08.920505 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-lkqqv\" (UniqueName: \"kubernetes.io/projected/c7fb916b-469a-46ca-806b-bd61c8b674fd-kube-api-access-lkqqv\") pod \"run-os-dataplane-edpm-2b9k2\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:08.920867 master-1 kubenswrapper[4771]: I1011 11:15:08.920725 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-ssh-key\") pod \"run-os-dataplane-edpm-2b9k2\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:08.926177 master-1 kubenswrapper[4771]: I1011 11:15:08.926112 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-ssh-key\") pod \"run-os-dataplane-edpm-2b9k2\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:08.926267 master-1 kubenswrapper[4771]: I1011 11:15:08.926191 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-inventory\") pod \"run-os-dataplane-edpm-2b9k2\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:08.941006 master-1 kubenswrapper[4771]: I1011 11:15:08.940942 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkqqv\" (UniqueName: \"kubernetes.io/projected/c7fb916b-469a-46ca-806b-bd61c8b674fd-kube-api-access-lkqqv\") pod \"run-os-dataplane-edpm-2b9k2\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:09.126605 master-1 kubenswrapper[4771]: I1011 11:15:09.126432 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:09.713994 master-1 kubenswrapper[4771]: I1011 11:15:09.713890 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/run-os-dataplane-edpm-2b9k2"] Oct 11 11:15:09.727752 master-1 kubenswrapper[4771]: I1011 11:15:09.727689 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 11:15:10.513780 master-1 kubenswrapper[4771]: I1011 11:15:10.513643 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/run-os-dataplane-edpm-2b9k2" event={"ID":"c7fb916b-469a-46ca-806b-bd61c8b674fd","Type":"ContainerStarted","Data":"15ef9683cf31a8e289e74a845c8cbc6879c65abafe501299d63d536a993eae47"} Oct 11 11:15:10.513780 master-1 kubenswrapper[4771]: I1011 11:15:10.513726 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/run-os-dataplane-edpm-2b9k2" event={"ID":"c7fb916b-469a-46ca-806b-bd61c8b674fd","Type":"ContainerStarted","Data":"f5d26dd462306ae00b18c5ee42db79300aa348307309be5e3086dc5621c421a3"} Oct 11 11:15:19.619587 master-1 kubenswrapper[4771]: I1011 11:15:19.619428 4771 generic.go:334] "Generic (PLEG): container finished" podID="c7fb916b-469a-46ca-806b-bd61c8b674fd" containerID="15ef9683cf31a8e289e74a845c8cbc6879c65abafe501299d63d536a993eae47" exitCode=0 Oct 11 11:15:19.619587 master-1 kubenswrapper[4771]: I1011 11:15:19.619542 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/run-os-dataplane-edpm-2b9k2" event={"ID":"c7fb916b-469a-46ca-806b-bd61c8b674fd","Type":"ContainerDied","Data":"15ef9683cf31a8e289e74a845c8cbc6879c65abafe501299d63d536a993eae47"} Oct 11 11:15:21.286982 master-1 kubenswrapper[4771]: I1011 11:15:21.286909 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:21.457763 master-1 kubenswrapper[4771]: I1011 11:15:21.457669 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-ssh-key\") pod \"c7fb916b-469a-46ca-806b-bd61c8b674fd\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " Oct 11 11:15:21.457763 master-1 kubenswrapper[4771]: I1011 11:15:21.457753 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkqqv\" (UniqueName: \"kubernetes.io/projected/c7fb916b-469a-46ca-806b-bd61c8b674fd-kube-api-access-lkqqv\") pod \"c7fb916b-469a-46ca-806b-bd61c8b674fd\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " Oct 11 11:15:21.458118 master-1 kubenswrapper[4771]: I1011 11:15:21.457814 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-inventory\") pod \"c7fb916b-469a-46ca-806b-bd61c8b674fd\" (UID: \"c7fb916b-469a-46ca-806b-bd61c8b674fd\") " Oct 11 11:15:21.471910 master-1 kubenswrapper[4771]: I1011 11:15:21.471758 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fb916b-469a-46ca-806b-bd61c8b674fd-kube-api-access-lkqqv" (OuterVolumeSpecName: "kube-api-access-lkqqv") pod "c7fb916b-469a-46ca-806b-bd61c8b674fd" (UID: "c7fb916b-469a-46ca-806b-bd61c8b674fd"). InnerVolumeSpecName "kube-api-access-lkqqv". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:15:21.501734 master-1 kubenswrapper[4771]: I1011 11:15:21.501649 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-inventory" (OuterVolumeSpecName: "inventory") pod "c7fb916b-469a-46ca-806b-bd61c8b674fd" (UID: "c7fb916b-469a-46ca-806b-bd61c8b674fd"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:15:21.514622 master-1 kubenswrapper[4771]: I1011 11:15:21.514532 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "c7fb916b-469a-46ca-806b-bd61c8b674fd" (UID: "c7fb916b-469a-46ca-806b-bd61c8b674fd"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:15:21.560979 master-1 kubenswrapper[4771]: I1011 11:15:21.560896 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:21.560979 master-1 kubenswrapper[4771]: I1011 11:15:21.560959 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-lkqqv\" (UniqueName: \"kubernetes.io/projected/c7fb916b-469a-46ca-806b-bd61c8b674fd-kube-api-access-lkqqv\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:21.560979 master-1 kubenswrapper[4771]: I1011 11:15:21.560983 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c7fb916b-469a-46ca-806b-bd61c8b674fd-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:21.660470 master-1 kubenswrapper[4771]: I1011 11:15:21.658260 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/run-os-dataplane-edpm-2b9k2" event={"ID":"c7fb916b-469a-46ca-806b-bd61c8b674fd","Type":"ContainerDied","Data":"f5d26dd462306ae00b18c5ee42db79300aa348307309be5e3086dc5621c421a3"} Oct 11 11:15:21.660470 master-1 kubenswrapper[4771]: I1011 11:15:21.658331 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d26dd462306ae00b18c5ee42db79300aa348307309be5e3086dc5621c421a3" Oct 11 11:15:21.660470 master-1 kubenswrapper[4771]: I1011 11:15:21.658413 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/run-os-dataplane-edpm-2b9k2" Oct 11 11:15:21.768793 master-1 kubenswrapper[4771]: I1011 11:15:21.768599 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/reboot-os-dataplane-edpm-nvx25"] Oct 11 11:15:21.769127 master-1 kubenswrapper[4771]: E1011 11:15:21.769070 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="c7fb916b-469a-46ca-806b-bd61c8b674fd" containerName="run-os-dataplane-edpm" Oct 11 11:15:21.769127 master-1 kubenswrapper[4771]: I1011 11:15:21.769091 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fb916b-469a-46ca-806b-bd61c8b674fd" containerName="run-os-dataplane-edpm" Oct 11 11:15:21.769427 master-1 kubenswrapper[4771]: I1011 11:15:21.769306 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="c7fb916b-469a-46ca-806b-bd61c8b674fd" containerName="run-os-dataplane-edpm" Oct 11 11:15:21.770304 master-1 kubenswrapper[4771]: I1011 11:15:21.770238 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:21.773607 master-1 kubenswrapper[4771]: I1011 11:15:21.773532 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:15:21.774076 master-1 kubenswrapper[4771]: I1011 11:15:21.774001 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:15:21.774735 master-1 kubenswrapper[4771]: I1011 11:15:21.774675 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:15:21.787441 master-1 kubenswrapper[4771]: I1011 11:15:21.787273 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/reboot-os-dataplane-edpm-nvx25"] Oct 11 11:15:21.971709 master-1 kubenswrapper[4771]: I1011 11:15:21.971588 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6qz\" (UniqueName: \"kubernetes.io/projected/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-kube-api-access-tg6qz\") pod \"reboot-os-dataplane-edpm-nvx25\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:21.972054 master-1 kubenswrapper[4771]: I1011 11:15:21.971827 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-inventory\") pod \"reboot-os-dataplane-edpm-nvx25\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:21.972054 master-1 kubenswrapper[4771]: I1011 11:15:21.971893 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-ssh-key\") pod \"reboot-os-dataplane-edpm-nvx25\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:22.073341 master-1 kubenswrapper[4771]: I1011 11:15:22.073242 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6qz\" (UniqueName: \"kubernetes.io/projected/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-kube-api-access-tg6qz\") pod \"reboot-os-dataplane-edpm-nvx25\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:22.073672 master-1 kubenswrapper[4771]: I1011 11:15:22.073423 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-inventory\") pod \"reboot-os-dataplane-edpm-nvx25\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:22.073672 master-1 kubenswrapper[4771]: I1011 11:15:22.073482 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-ssh-key\") pod \"reboot-os-dataplane-edpm-nvx25\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:22.077697 master-1 kubenswrapper[4771]: I1011 11:15:22.077633 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-ssh-key\") pod \"reboot-os-dataplane-edpm-nvx25\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:22.080014 master-1 kubenswrapper[4771]: I1011 11:15:22.079930 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-inventory\") pod \"reboot-os-dataplane-edpm-nvx25\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:22.107326 master-1 kubenswrapper[4771]: I1011 11:15:22.107206 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6qz\" (UniqueName: \"kubernetes.io/projected/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-kube-api-access-tg6qz\") pod \"reboot-os-dataplane-edpm-nvx25\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:22.159718 master-1 kubenswrapper[4771]: I1011 11:15:22.159651 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:15:22.852210 master-1 kubenswrapper[4771]: I1011 11:15:22.851833 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/run-os-networker-deploy-networkers-8hnn8"] Oct 11 11:15:22.857929 master-1 kubenswrapper[4771]: I1011 11:15:22.857430 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:22.862561 master-1 kubenswrapper[4771]: I1011 11:15:22.861907 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-networkers" Oct 11 11:15:22.871727 master-1 kubenswrapper[4771]: I1011 11:15:22.871658 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/reboot-os-dataplane-edpm-nvx25"] Oct 11 11:15:22.882108 master-1 kubenswrapper[4771]: I1011 11:15:22.881955 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/run-os-networker-deploy-networkers-8hnn8"] Oct 11 11:15:23.006082 master-1 kubenswrapper[4771]: I1011 11:15:23.005982 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-inventory\") pod \"run-os-networker-deploy-networkers-8hnn8\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.006373 master-1 kubenswrapper[4771]: I1011 11:15:23.006104 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255x2\" (UniqueName: \"kubernetes.io/projected/399d2deb-21cc-4976-ae9e-170fa7d754b2-kube-api-access-255x2\") pod \"run-os-networker-deploy-networkers-8hnn8\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.006373 master-1 kubenswrapper[4771]: I1011 11:15:23.006176 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-ssh-key\") pod \"run-os-networker-deploy-networkers-8hnn8\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.110518 master-1 kubenswrapper[4771]: I1011 11:15:23.110278 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-inventory\") pod \"run-os-networker-deploy-networkers-8hnn8\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.110518 master-1 kubenswrapper[4771]: I1011 11:15:23.110456 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-255x2\" (UniqueName: \"kubernetes.io/projected/399d2deb-21cc-4976-ae9e-170fa7d754b2-kube-api-access-255x2\") pod \"run-os-networker-deploy-networkers-8hnn8\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.110980 master-1 kubenswrapper[4771]: I1011 11:15:23.110541 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-ssh-key\") pod \"run-os-networker-deploy-networkers-8hnn8\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.115093 master-1 kubenswrapper[4771]: I1011 11:15:23.115021 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-inventory\") pod \"run-os-networker-deploy-networkers-8hnn8\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.116295 master-1 kubenswrapper[4771]: I1011 11:15:23.116235 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-ssh-key\") pod \"run-os-networker-deploy-networkers-8hnn8\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.129937 master-1 kubenswrapper[4771]: I1011 11:15:23.129850 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-255x2\" (UniqueName: \"kubernetes.io/projected/399d2deb-21cc-4976-ae9e-170fa7d754b2-kube-api-access-255x2\") pod \"run-os-networker-deploy-networkers-8hnn8\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.223605 master-1 kubenswrapper[4771]: I1011 11:15:23.223500 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:23.698510 master-1 kubenswrapper[4771]: I1011 11:15:23.698394 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/reboot-os-dataplane-edpm-nvx25" event={"ID":"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff","Type":"ContainerStarted","Data":"7dad6181f616c54767b9aa28947089dd8d2181c28cfd68675c9cdbe33f1d675b"} Oct 11 11:15:23.698820 master-1 kubenswrapper[4771]: I1011 11:15:23.698518 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/reboot-os-dataplane-edpm-nvx25" event={"ID":"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff","Type":"ContainerStarted","Data":"38d65c631695208ab4af38ed5f867d266e1a6cf55f11c5f792930a86a2ba576d"} Oct 11 11:15:23.725612 master-1 kubenswrapper[4771]: I1011 11:15:23.725468 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/reboot-os-dataplane-edpm-nvx25" podStartSLOduration=2.196868979 podStartE2EDuration="2.725440853s" podCreationTimestamp="2025-10-11 11:15:21 +0000 UTC" firstStartedPulling="2025-10-11 11:15:22.856959959 +0000 UTC m=+2954.831186410" lastFinishedPulling="2025-10-11 11:15:23.385531843 +0000 UTC m=+2955.359758284" observedRunningTime="2025-10-11 11:15:23.724479075 +0000 UTC m=+2955.698705526" watchObservedRunningTime="2025-10-11 11:15:23.725440853 +0000 UTC m=+2955.699667294" Oct 11 11:15:23.845951 master-1 kubenswrapper[4771]: W1011 11:15:23.844123 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod399d2deb_21cc_4976_ae9e_170fa7d754b2.slice/crio-203f88f268696cc6105fcf08ae1d25636aca0628e0290a26f22d7381ec854ebb WatchSource:0}: Error finding container 203f88f268696cc6105fcf08ae1d25636aca0628e0290a26f22d7381ec854ebb: Status 404 returned error can't find the container with id 203f88f268696cc6105fcf08ae1d25636aca0628e0290a26f22d7381ec854ebb Oct 11 11:15:23.845951 master-1 kubenswrapper[4771]: I1011 11:15:23.844157 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/run-os-networker-deploy-networkers-8hnn8"] Oct 11 11:15:24.716024 master-1 kubenswrapper[4771]: I1011 11:15:24.715900 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/run-os-networker-deploy-networkers-8hnn8" event={"ID":"399d2deb-21cc-4976-ae9e-170fa7d754b2","Type":"ContainerStarted","Data":"d09b8107b6076b69407115dd79fa8a732a65779488ad44540ab90f20b2f25240"} Oct 11 11:15:24.717223 master-1 kubenswrapper[4771]: I1011 11:15:24.716029 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/run-os-networker-deploy-networkers-8hnn8" event={"ID":"399d2deb-21cc-4976-ae9e-170fa7d754b2","Type":"ContainerStarted","Data":"203f88f268696cc6105fcf08ae1d25636aca0628e0290a26f22d7381ec854ebb"} Oct 11 11:15:24.746959 master-1 kubenswrapper[4771]: I1011 11:15:24.746616 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/run-os-networker-deploy-networkers-8hnn8" podStartSLOduration=2.313171919 podStartE2EDuration="2.746589092s" podCreationTimestamp="2025-10-11 11:15:22 +0000 UTC" firstStartedPulling="2025-10-11 11:15:23.847297177 +0000 UTC m=+2955.821523618" lastFinishedPulling="2025-10-11 11:15:24.28071435 +0000 UTC m=+2956.254940791" observedRunningTime="2025-10-11 11:15:24.74442157 +0000 UTC m=+2956.718648031" watchObservedRunningTime="2025-10-11 11:15:24.746589092 +0000 UTC m=+2956.720815573" Oct 11 11:15:25.323135 master-1 kubenswrapper[4771]: I1011 11:15:25.323047 4771 scope.go:117] "RemoveContainer" containerID="ea50bb78d4de53e43e9be3f2830fede428957c124838ed0305c9a99b641c0252" Oct 11 11:15:34.829097 master-1 kubenswrapper[4771]: I1011 11:15:34.829002 4771 generic.go:334] "Generic (PLEG): container finished" podID="399d2deb-21cc-4976-ae9e-170fa7d754b2" containerID="d09b8107b6076b69407115dd79fa8a732a65779488ad44540ab90f20b2f25240" exitCode=0 Oct 11 11:15:34.829097 master-1 kubenswrapper[4771]: I1011 11:15:34.829078 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/run-os-networker-deploy-networkers-8hnn8" event={"ID":"399d2deb-21cc-4976-ae9e-170fa7d754b2","Type":"ContainerDied","Data":"d09b8107b6076b69407115dd79fa8a732a65779488ad44540ab90f20b2f25240"} Oct 11 11:15:36.447501 master-1 kubenswrapper[4771]: I1011 11:15:36.447436 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:36.570344 master-1 kubenswrapper[4771]: I1011 11:15:36.570103 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-255x2\" (UniqueName: \"kubernetes.io/projected/399d2deb-21cc-4976-ae9e-170fa7d754b2-kube-api-access-255x2\") pod \"399d2deb-21cc-4976-ae9e-170fa7d754b2\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " Oct 11 11:15:36.570344 master-1 kubenswrapper[4771]: I1011 11:15:36.570236 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-inventory\") pod \"399d2deb-21cc-4976-ae9e-170fa7d754b2\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " Oct 11 11:15:36.571490 master-1 kubenswrapper[4771]: I1011 11:15:36.571113 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-ssh-key\") pod \"399d2deb-21cc-4976-ae9e-170fa7d754b2\" (UID: \"399d2deb-21cc-4976-ae9e-170fa7d754b2\") " Oct 11 11:15:36.573578 master-1 kubenswrapper[4771]: I1011 11:15:36.573522 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399d2deb-21cc-4976-ae9e-170fa7d754b2-kube-api-access-255x2" (OuterVolumeSpecName: "kube-api-access-255x2") pod "399d2deb-21cc-4976-ae9e-170fa7d754b2" (UID: "399d2deb-21cc-4976-ae9e-170fa7d754b2"). InnerVolumeSpecName "kube-api-access-255x2". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:15:36.594062 master-1 kubenswrapper[4771]: I1011 11:15:36.593760 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "399d2deb-21cc-4976-ae9e-170fa7d754b2" (UID: "399d2deb-21cc-4976-ae9e-170fa7d754b2"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:15:36.594456 master-1 kubenswrapper[4771]: I1011 11:15:36.594383 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-inventory" (OuterVolumeSpecName: "inventory") pod "399d2deb-21cc-4976-ae9e-170fa7d754b2" (UID: "399d2deb-21cc-4976-ae9e-170fa7d754b2"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:15:36.673409 master-1 kubenswrapper[4771]: I1011 11:15:36.673330 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-255x2\" (UniqueName: \"kubernetes.io/projected/399d2deb-21cc-4976-ae9e-170fa7d754b2-kube-api-access-255x2\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:36.673409 master-1 kubenswrapper[4771]: I1011 11:15:36.673394 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:36.673409 master-1 kubenswrapper[4771]: I1011 11:15:36.673406 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/399d2deb-21cc-4976-ae9e-170fa7d754b2-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:15:36.860332 master-1 kubenswrapper[4771]: I1011 11:15:36.860114 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/run-os-networker-deploy-networkers-8hnn8" event={"ID":"399d2deb-21cc-4976-ae9e-170fa7d754b2","Type":"ContainerDied","Data":"203f88f268696cc6105fcf08ae1d25636aca0628e0290a26f22d7381ec854ebb"} Oct 11 11:15:36.860332 master-1 kubenswrapper[4771]: I1011 11:15:36.860173 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="203f88f268696cc6105fcf08ae1d25636aca0628e0290a26f22d7381ec854ebb" Oct 11 11:15:36.860332 master-1 kubenswrapper[4771]: I1011 11:15:36.860245 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/run-os-networker-deploy-networkers-8hnn8" Oct 11 11:15:36.993427 master-1 kubenswrapper[4771]: I1011 11:15:36.992618 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/reboot-os-networker-deploy-networkers-fgsg5"] Oct 11 11:15:36.993427 master-1 kubenswrapper[4771]: E1011 11:15:36.993088 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="399d2deb-21cc-4976-ae9e-170fa7d754b2" containerName="run-os-networker-deploy-networkers" Oct 11 11:15:36.993427 master-1 kubenswrapper[4771]: I1011 11:15:36.993111 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="399d2deb-21cc-4976-ae9e-170fa7d754b2" containerName="run-os-networker-deploy-networkers" Oct 11 11:15:36.993427 master-1 kubenswrapper[4771]: I1011 11:15:36.993395 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="399d2deb-21cc-4976-ae9e-170fa7d754b2" containerName="run-os-networker-deploy-networkers" Oct 11 11:15:36.995518 master-1 kubenswrapper[4771]: I1011 11:15:36.994633 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:36.997595 master-1 kubenswrapper[4771]: I1011 11:15:36.997558 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-networkers" Oct 11 11:15:37.040056 master-1 kubenswrapper[4771]: I1011 11:15:37.039937 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/reboot-os-networker-deploy-networkers-fgsg5"] Oct 11 11:15:37.083842 master-1 kubenswrapper[4771]: E1011 11:15:37.083751 4771 cadvisor_stats_provider.go:516] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod399d2deb_21cc_4976_ae9e_170fa7d754b2.slice\": RecentStats: unable to find data in memory cache]" Oct 11 11:15:37.182079 master-1 kubenswrapper[4771]: I1011 11:15:37.181842 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-ssh-key\") pod \"reboot-os-networker-deploy-networkers-fgsg5\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:37.182079 master-1 kubenswrapper[4771]: I1011 11:15:37.181948 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-inventory\") pod \"reboot-os-networker-deploy-networkers-fgsg5\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:37.182079 master-1 kubenswrapper[4771]: I1011 11:15:37.182036 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdnn\" (UniqueName: \"kubernetes.io/projected/fa11837b-6428-471c-b8d4-ffafd00d954b-kube-api-access-5vdnn\") pod \"reboot-os-networker-deploy-networkers-fgsg5\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:37.284529 master-1 kubenswrapper[4771]: I1011 11:15:37.284444 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdnn\" (UniqueName: \"kubernetes.io/projected/fa11837b-6428-471c-b8d4-ffafd00d954b-kube-api-access-5vdnn\") pod \"reboot-os-networker-deploy-networkers-fgsg5\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:37.284959 master-1 kubenswrapper[4771]: I1011 11:15:37.284740 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-ssh-key\") pod \"reboot-os-networker-deploy-networkers-fgsg5\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:37.284959 master-1 kubenswrapper[4771]: I1011 11:15:37.284791 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-inventory\") pod \"reboot-os-networker-deploy-networkers-fgsg5\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:37.292845 master-1 kubenswrapper[4771]: I1011 11:15:37.292804 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-ssh-key\") pod \"reboot-os-networker-deploy-networkers-fgsg5\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:37.293083 master-1 kubenswrapper[4771]: I1011 11:15:37.293046 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-inventory\") pod \"reboot-os-networker-deploy-networkers-fgsg5\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:37.320541 master-1 kubenswrapper[4771]: I1011 11:15:37.320469 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdnn\" (UniqueName: \"kubernetes.io/projected/fa11837b-6428-471c-b8d4-ffafd00d954b-kube-api-access-5vdnn\") pod \"reboot-os-networker-deploy-networkers-fgsg5\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:37.618933 master-1 kubenswrapper[4771]: I1011 11:15:37.618850 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:15:38.186240 master-1 kubenswrapper[4771]: I1011 11:15:38.186036 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/reboot-os-networker-deploy-networkers-fgsg5"] Oct 11 11:15:38.192285 master-1 kubenswrapper[4771]: W1011 11:15:38.192209 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa11837b_6428_471c_b8d4_ffafd00d954b.slice/crio-7710262e3855b3c019a49b8d470f18483fe3fa4ea05080e4da182e7e68559971 WatchSource:0}: Error finding container 7710262e3855b3c019a49b8d470f18483fe3fa4ea05080e4da182e7e68559971: Status 404 returned error can't find the container with id 7710262e3855b3c019a49b8d470f18483fe3fa4ea05080e4da182e7e68559971 Oct 11 11:15:38.882404 master-1 kubenswrapper[4771]: I1011 11:15:38.882192 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" event={"ID":"fa11837b-6428-471c-b8d4-ffafd00d954b","Type":"ContainerStarted","Data":"7710262e3855b3c019a49b8d470f18483fe3fa4ea05080e4da182e7e68559971"} Oct 11 11:15:39.895220 master-1 kubenswrapper[4771]: I1011 11:15:39.895118 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" event={"ID":"fa11837b-6428-471c-b8d4-ffafd00d954b","Type":"ContainerStarted","Data":"89933431cf76a9a43f09580949f86bd5d879200bed420eae4edc172dd83c04be"} Oct 11 11:15:39.927400 master-1 kubenswrapper[4771]: I1011 11:15:39.927240 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" podStartSLOduration=3.49739094 podStartE2EDuration="3.927216s" podCreationTimestamp="2025-10-11 11:15:36 +0000 UTC" firstStartedPulling="2025-10-11 11:15:38.194962368 +0000 UTC m=+2970.169188809" lastFinishedPulling="2025-10-11 11:15:38.624787428 +0000 UTC m=+2970.599013869" observedRunningTime="2025-10-11 11:15:39.926573572 +0000 UTC m=+2971.900800023" watchObservedRunningTime="2025-10-11 11:15:39.927216 +0000 UTC m=+2971.901442441" Oct 11 11:16:18.351344 master-1 kubenswrapper[4771]: I1011 11:16:18.351211 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/community-operators-dzcrl"] Oct 11 11:16:18.354569 master-1 kubenswrapper[4771]: I1011 11:16:18.354536 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.403272 master-1 kubenswrapper[4771]: I1011 11:16:18.403170 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-dzcrl"] Oct 11 11:16:18.451567 master-1 kubenswrapper[4771]: I1011 11:16:18.451474 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4smm\" (UniqueName: \"kubernetes.io/projected/34b4de24-eb85-4793-9435-44bc5c82aab9-kube-api-access-k4smm\") pod \"community-operators-dzcrl\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.451567 master-1 kubenswrapper[4771]: I1011 11:16:18.451547 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-catalog-content\") pod \"community-operators-dzcrl\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.452124 master-1 kubenswrapper[4771]: I1011 11:16:18.451952 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-utilities\") pod \"community-operators-dzcrl\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.556025 master-1 kubenswrapper[4771]: I1011 11:16:18.555872 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-k4smm\" (UniqueName: \"kubernetes.io/projected/34b4de24-eb85-4793-9435-44bc5c82aab9-kube-api-access-k4smm\") pod \"community-operators-dzcrl\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.556025 master-1 kubenswrapper[4771]: I1011 11:16:18.555936 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-catalog-content\") pod \"community-operators-dzcrl\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.556025 master-1 kubenswrapper[4771]: I1011 11:16:18.556038 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-utilities\") pod \"community-operators-dzcrl\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.556599 master-1 kubenswrapper[4771]: I1011 11:16:18.556573 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-utilities\") pod \"community-operators-dzcrl\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.557135 master-1 kubenswrapper[4771]: I1011 11:16:18.557082 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-catalog-content\") pod \"community-operators-dzcrl\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.584813 master-1 kubenswrapper[4771]: I1011 11:16:18.584709 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4smm\" (UniqueName: \"kubernetes.io/projected/34b4de24-eb85-4793-9435-44bc5c82aab9-kube-api-access-k4smm\") pod \"community-operators-dzcrl\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:18.724671 master-1 kubenswrapper[4771]: I1011 11:16:18.724462 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:19.230397 master-1 kubenswrapper[4771]: W1011 11:16:19.229812 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b4de24_eb85_4793_9435_44bc5c82aab9.slice/crio-40d10a341484c58e4a87284c38624488c83e1dec77bc2337e7b4e3223c2bf0ce WatchSource:0}: Error finding container 40d10a341484c58e4a87284c38624488c83e1dec77bc2337e7b4e3223c2bf0ce: Status 404 returned error can't find the container with id 40d10a341484c58e4a87284c38624488c83e1dec77bc2337e7b4e3223c2bf0ce Oct 11 11:16:19.233436 master-1 kubenswrapper[4771]: I1011 11:16:19.233340 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-dzcrl"] Oct 11 11:16:19.330146 master-1 kubenswrapper[4771]: I1011 11:16:19.330048 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-dzcrl" event={"ID":"34b4de24-eb85-4793-9435-44bc5c82aab9","Type":"ContainerStarted","Data":"40d10a341484c58e4a87284c38624488c83e1dec77bc2337e7b4e3223c2bf0ce"} Oct 11 11:16:20.342807 master-1 kubenswrapper[4771]: I1011 11:16:20.342725 4771 generic.go:334] "Generic (PLEG): container finished" podID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerID="4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846" exitCode=0 Oct 11 11:16:20.342807 master-1 kubenswrapper[4771]: I1011 11:16:20.342807 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-dzcrl" event={"ID":"34b4de24-eb85-4793-9435-44bc5c82aab9","Type":"ContainerDied","Data":"4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846"} Oct 11 11:16:21.354443 master-1 kubenswrapper[4771]: I1011 11:16:21.354332 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-dzcrl" event={"ID":"34b4de24-eb85-4793-9435-44bc5c82aab9","Type":"ContainerStarted","Data":"7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb"} Oct 11 11:16:22.367499 master-1 kubenswrapper[4771]: I1011 11:16:22.367410 4771 generic.go:334] "Generic (PLEG): container finished" podID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerID="7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb" exitCode=0 Oct 11 11:16:22.368536 master-1 kubenswrapper[4771]: I1011 11:16:22.367508 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-dzcrl" event={"ID":"34b4de24-eb85-4793-9435-44bc5c82aab9","Type":"ContainerDied","Data":"7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb"} Oct 11 11:16:23.379150 master-1 kubenswrapper[4771]: I1011 11:16:23.379016 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-dzcrl" event={"ID":"34b4de24-eb85-4793-9435-44bc5c82aab9","Type":"ContainerStarted","Data":"7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad"} Oct 11 11:16:23.413277 master-1 kubenswrapper[4771]: I1011 11:16:23.413181 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/community-operators-dzcrl" podStartSLOduration=2.969984584 podStartE2EDuration="5.413155465s" podCreationTimestamp="2025-10-11 11:16:18 +0000 UTC" firstStartedPulling="2025-10-11 11:16:20.345516488 +0000 UTC m=+3012.319742939" lastFinishedPulling="2025-10-11 11:16:22.788687349 +0000 UTC m=+3014.762913820" observedRunningTime="2025-10-11 11:16:23.406986038 +0000 UTC m=+3015.381212489" watchObservedRunningTime="2025-10-11 11:16:23.413155465 +0000 UTC m=+3015.387381926" Oct 11 11:16:28.725795 master-1 kubenswrapper[4771]: I1011 11:16:28.725691 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:28.725795 master-1 kubenswrapper[4771]: I1011 11:16:28.725808 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:28.805118 master-1 kubenswrapper[4771]: I1011 11:16:28.804990 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:29.502764 master-1 kubenswrapper[4771]: I1011 11:16:29.502696 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:29.591189 master-1 kubenswrapper[4771]: I1011 11:16:29.591111 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-dzcrl"] Oct 11 11:16:31.467723 master-1 kubenswrapper[4771]: I1011 11:16:31.467628 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/community-operators-dzcrl" podUID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerName="registry-server" containerID="cri-o://7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad" gracePeriod=2 Oct 11 11:16:32.105050 master-1 kubenswrapper[4771]: I1011 11:16:32.104981 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:32.195037 master-1 kubenswrapper[4771]: I1011 11:16:32.194956 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-utilities\") pod \"34b4de24-eb85-4793-9435-44bc5c82aab9\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " Oct 11 11:16:32.195341 master-1 kubenswrapper[4771]: I1011 11:16:32.195132 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-catalog-content\") pod \"34b4de24-eb85-4793-9435-44bc5c82aab9\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " Oct 11 11:16:32.195341 master-1 kubenswrapper[4771]: I1011 11:16:32.195203 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4smm\" (UniqueName: \"kubernetes.io/projected/34b4de24-eb85-4793-9435-44bc5c82aab9-kube-api-access-k4smm\") pod \"34b4de24-eb85-4793-9435-44bc5c82aab9\" (UID: \"34b4de24-eb85-4793-9435-44bc5c82aab9\") " Oct 11 11:16:32.197374 master-1 kubenswrapper[4771]: I1011 11:16:32.197312 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-utilities" (OuterVolumeSpecName: "utilities") pod "34b4de24-eb85-4793-9435-44bc5c82aab9" (UID: "34b4de24-eb85-4793-9435-44bc5c82aab9"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:16:32.199814 master-1 kubenswrapper[4771]: I1011 11:16:32.199720 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b4de24-eb85-4793-9435-44bc5c82aab9-kube-api-access-k4smm" (OuterVolumeSpecName: "kube-api-access-k4smm") pod "34b4de24-eb85-4793-9435-44bc5c82aab9" (UID: "34b4de24-eb85-4793-9435-44bc5c82aab9"). InnerVolumeSpecName "kube-api-access-k4smm". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:16:32.300520 master-1 kubenswrapper[4771]: I1011 11:16:32.300335 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-k4smm\" (UniqueName: \"kubernetes.io/projected/34b4de24-eb85-4793-9435-44bc5c82aab9-kube-api-access-k4smm\") on node \"master-1\" DevicePath \"\"" Oct 11 11:16:32.300520 master-1 kubenswrapper[4771]: I1011 11:16:32.300419 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:16:32.320967 master-1 kubenswrapper[4771]: I1011 11:16:32.320885 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "34b4de24-eb85-4793-9435-44bc5c82aab9" (UID: "34b4de24-eb85-4793-9435-44bc5c82aab9"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:16:32.402582 master-1 kubenswrapper[4771]: I1011 11:16:32.402482 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/34b4de24-eb85-4793-9435-44bc5c82aab9-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:16:32.482567 master-1 kubenswrapper[4771]: I1011 11:16:32.482333 4771 generic.go:334] "Generic (PLEG): container finished" podID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerID="7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad" exitCode=0 Oct 11 11:16:32.482567 master-1 kubenswrapper[4771]: I1011 11:16:32.482401 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-dzcrl" event={"ID":"34b4de24-eb85-4793-9435-44bc5c82aab9","Type":"ContainerDied","Data":"7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad"} Oct 11 11:16:32.482567 master-1 kubenswrapper[4771]: I1011 11:16:32.482455 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-dzcrl" event={"ID":"34b4de24-eb85-4793-9435-44bc5c82aab9","Type":"ContainerDied","Data":"40d10a341484c58e4a87284c38624488c83e1dec77bc2337e7b4e3223c2bf0ce"} Oct 11 11:16:32.482567 master-1 kubenswrapper[4771]: I1011 11:16:32.482488 4771 scope.go:117] "RemoveContainer" containerID="7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad" Oct 11 11:16:32.483225 master-1 kubenswrapper[4771]: I1011 11:16:32.482728 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-dzcrl" Oct 11 11:16:32.513397 master-1 kubenswrapper[4771]: I1011 11:16:32.513284 4771 scope.go:117] "RemoveContainer" containerID="7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb" Oct 11 11:16:32.555433 master-1 kubenswrapper[4771]: I1011 11:16:32.555301 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-dzcrl"] Oct 11 11:16:32.562056 master-1 kubenswrapper[4771]: I1011 11:16:32.562000 4771 scope.go:117] "RemoveContainer" containerID="4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846" Oct 11 11:16:32.567575 master-1 kubenswrapper[4771]: I1011 11:16:32.567521 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/community-operators-dzcrl"] Oct 11 11:16:32.606961 master-1 kubenswrapper[4771]: I1011 11:16:32.606667 4771 scope.go:117] "RemoveContainer" containerID="7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad" Oct 11 11:16:32.607317 master-1 kubenswrapper[4771]: E1011 11:16:32.607274 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad\": container with ID starting with 7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad not found: ID does not exist" containerID="7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad" Oct 11 11:16:32.607460 master-1 kubenswrapper[4771]: I1011 11:16:32.607326 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad"} err="failed to get container status \"7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad\": rpc error: code = NotFound desc = could not find container \"7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad\": container with ID starting with 7554b4551b7e1efbf0288693c2eee8e8636ee8f4c5e8b6b631bbdca7cc40f3ad not found: ID does not exist" Oct 11 11:16:32.607460 master-1 kubenswrapper[4771]: I1011 11:16:32.607370 4771 scope.go:117] "RemoveContainer" containerID="7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb" Oct 11 11:16:32.608145 master-1 kubenswrapper[4771]: E1011 11:16:32.607894 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb\": container with ID starting with 7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb not found: ID does not exist" containerID="7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb" Oct 11 11:16:32.608145 master-1 kubenswrapper[4771]: I1011 11:16:32.607954 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb"} err="failed to get container status \"7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb\": rpc error: code = NotFound desc = could not find container \"7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb\": container with ID starting with 7d791a7487548b9eff666906ee0b8402629e31ea95f7d2a171992e5a10d919fb not found: ID does not exist" Oct 11 11:16:32.608145 master-1 kubenswrapper[4771]: I1011 11:16:32.607999 4771 scope.go:117] "RemoveContainer" containerID="4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846" Oct 11 11:16:32.608636 master-1 kubenswrapper[4771]: E1011 11:16:32.608575 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846\": container with ID starting with 4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846 not found: ID does not exist" containerID="4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846" Oct 11 11:16:32.608721 master-1 kubenswrapper[4771]: I1011 11:16:32.608641 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846"} err="failed to get container status \"4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846\": rpc error: code = NotFound desc = could not find container \"4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846\": container with ID starting with 4569c17a93983fc7c9806039d762a1c0353f579ec1e297c01f1597f9f52f1846 not found: ID does not exist" Oct 11 11:16:34.455913 master-1 kubenswrapper[4771]: I1011 11:16:34.455795 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34b4de24-eb85-4793-9435-44bc5c82aab9" path="/var/lib/kubelet/pods/34b4de24-eb85-4793-9435-44bc5c82aab9/volumes" Oct 11 11:16:38.571196 master-1 kubenswrapper[4771]: I1011 11:16:38.571098 4771 generic.go:334] "Generic (PLEG): container finished" podID="c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff" containerID="7dad6181f616c54767b9aa28947089dd8d2181c28cfd68675c9cdbe33f1d675b" exitCode=0 Oct 11 11:16:38.571196 master-1 kubenswrapper[4771]: I1011 11:16:38.571163 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/reboot-os-dataplane-edpm-nvx25" event={"ID":"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff","Type":"ContainerDied","Data":"7dad6181f616c54767b9aa28947089dd8d2181c28cfd68675c9cdbe33f1d675b"} Oct 11 11:16:40.300306 master-1 kubenswrapper[4771]: I1011 11:16:40.300225 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:16:40.406598 master-1 kubenswrapper[4771]: I1011 11:16:40.406342 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg6qz\" (UniqueName: \"kubernetes.io/projected/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-kube-api-access-tg6qz\") pod \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " Oct 11 11:16:40.407017 master-1 kubenswrapper[4771]: I1011 11:16:40.406761 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-inventory\") pod \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " Oct 11 11:16:40.407017 master-1 kubenswrapper[4771]: I1011 11:16:40.406824 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-ssh-key\") pod \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\" (UID: \"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff\") " Oct 11 11:16:40.412726 master-1 kubenswrapper[4771]: I1011 11:16:40.412625 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-kube-api-access-tg6qz" (OuterVolumeSpecName: "kube-api-access-tg6qz") pod "c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff" (UID: "c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff"). InnerVolumeSpecName "kube-api-access-tg6qz". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:16:40.430670 master-1 kubenswrapper[4771]: I1011 11:16:40.430604 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff" (UID: "c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:16:40.455698 master-1 kubenswrapper[4771]: I1011 11:16:40.455483 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-inventory" (OuterVolumeSpecName: "inventory") pod "c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff" (UID: "c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:16:40.511148 master-1 kubenswrapper[4771]: I1011 11:16:40.511047 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-tg6qz\" (UniqueName: \"kubernetes.io/projected/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-kube-api-access-tg6qz\") on node \"master-1\" DevicePath \"\"" Oct 11 11:16:40.511148 master-1 kubenswrapper[4771]: I1011 11:16:40.511119 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:16:40.511148 master-1 kubenswrapper[4771]: I1011 11:16:40.511134 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:16:40.603399 master-1 kubenswrapper[4771]: I1011 11:16:40.603284 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/reboot-os-dataplane-edpm-nvx25" event={"ID":"c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff","Type":"ContainerDied","Data":"38d65c631695208ab4af38ed5f867d266e1a6cf55f11c5f792930a86a2ba576d"} Oct 11 11:16:40.603399 master-1 kubenswrapper[4771]: I1011 11:16:40.603382 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/reboot-os-dataplane-edpm-nvx25" Oct 11 11:16:40.603399 master-1 kubenswrapper[4771]: I1011 11:16:40.603415 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38d65c631695208ab4af38ed5f867d266e1a6cf55f11c5f792930a86a2ba576d" Oct 11 11:16:55.767324 master-1 kubenswrapper[4771]: I1011 11:16:55.767102 4771 generic.go:334] "Generic (PLEG): container finished" podID="fa11837b-6428-471c-b8d4-ffafd00d954b" containerID="89933431cf76a9a43f09580949f86bd5d879200bed420eae4edc172dd83c04be" exitCode=0 Oct 11 11:16:55.767324 master-1 kubenswrapper[4771]: I1011 11:16:55.767212 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" event={"ID":"fa11837b-6428-471c-b8d4-ffafd00d954b","Type":"ContainerDied","Data":"89933431cf76a9a43f09580949f86bd5d879200bed420eae4edc172dd83c04be"} Oct 11 11:16:57.465770 master-1 kubenswrapper[4771]: I1011 11:16:57.465701 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:16:57.523046 master-1 kubenswrapper[4771]: I1011 11:16:57.522949 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vdnn\" (UniqueName: \"kubernetes.io/projected/fa11837b-6428-471c-b8d4-ffafd00d954b-kube-api-access-5vdnn\") pod \"fa11837b-6428-471c-b8d4-ffafd00d954b\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " Oct 11 11:16:57.523522 master-1 kubenswrapper[4771]: I1011 11:16:57.523493 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-ssh-key\") pod \"fa11837b-6428-471c-b8d4-ffafd00d954b\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " Oct 11 11:16:57.523775 master-1 kubenswrapper[4771]: I1011 11:16:57.523749 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-inventory\") pod \"fa11837b-6428-471c-b8d4-ffafd00d954b\" (UID: \"fa11837b-6428-471c-b8d4-ffafd00d954b\") " Oct 11 11:16:57.527868 master-1 kubenswrapper[4771]: I1011 11:16:57.527799 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa11837b-6428-471c-b8d4-ffafd00d954b-kube-api-access-5vdnn" (OuterVolumeSpecName: "kube-api-access-5vdnn") pod "fa11837b-6428-471c-b8d4-ffafd00d954b" (UID: "fa11837b-6428-471c-b8d4-ffafd00d954b"). InnerVolumeSpecName "kube-api-access-5vdnn". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:16:57.550948 master-1 kubenswrapper[4771]: I1011 11:16:57.550874 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-inventory" (OuterVolumeSpecName: "inventory") pod "fa11837b-6428-471c-b8d4-ffafd00d954b" (UID: "fa11837b-6428-471c-b8d4-ffafd00d954b"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:16:57.551298 master-1 kubenswrapper[4771]: I1011 11:16:57.551227 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "fa11837b-6428-471c-b8d4-ffafd00d954b" (UID: "fa11837b-6428-471c-b8d4-ffafd00d954b"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:16:57.626230 master-1 kubenswrapper[4771]: I1011 11:16:57.626154 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:16:57.626230 master-1 kubenswrapper[4771]: I1011 11:16:57.626202 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-5vdnn\" (UniqueName: \"kubernetes.io/projected/fa11837b-6428-471c-b8d4-ffafd00d954b-kube-api-access-5vdnn\") on node \"master-1\" DevicePath \"\"" Oct 11 11:16:57.626230 master-1 kubenswrapper[4771]: I1011 11:16:57.626213 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fa11837b-6428-471c-b8d4-ffafd00d954b-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:16:57.790256 master-1 kubenswrapper[4771]: I1011 11:16:57.790178 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" event={"ID":"fa11837b-6428-471c-b8d4-ffafd00d954b","Type":"ContainerDied","Data":"7710262e3855b3c019a49b8d470f18483fe3fa4ea05080e4da182e7e68559971"} Oct 11 11:16:57.790256 master-1 kubenswrapper[4771]: I1011 11:16:57.790232 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7710262e3855b3c019a49b8d470f18483fe3fa4ea05080e4da182e7e68559971" Oct 11 11:16:57.790719 master-1 kubenswrapper[4771]: I1011 11:16:57.790277 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/reboot-os-networker-deploy-networkers-fgsg5" Oct 11 11:16:57.940635 master-1 kubenswrapper[4771]: I1011 11:16:57.940462 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/install-certs-networker-deploy-networkers-nv56q"] Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: E1011 11:16:57.940919 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerName="extract-content" Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: I1011 11:16:57.940948 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerName="extract-content" Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: E1011 11:16:57.940972 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerName="registry-server" Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: I1011 11:16:57.940984 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerName="registry-server" Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: E1011 11:16:57.941004 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerName="extract-utilities" Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: I1011 11:16:57.941016 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerName="extract-utilities" Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: E1011 11:16:57.941037 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff" containerName="reboot-os-dataplane-edpm" Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: I1011 11:16:57.941046 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff" containerName="reboot-os-dataplane-edpm" Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: E1011 11:16:57.941071 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="fa11837b-6428-471c-b8d4-ffafd00d954b" containerName="reboot-os-networker-deploy-networkers" Oct 11 11:16:57.941146 master-1 kubenswrapper[4771]: I1011 11:16:57.941081 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa11837b-6428-471c-b8d4-ffafd00d954b" containerName="reboot-os-networker-deploy-networkers" Oct 11 11:16:57.941494 master-1 kubenswrapper[4771]: I1011 11:16:57.941319 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="34b4de24-eb85-4793-9435-44bc5c82aab9" containerName="registry-server" Oct 11 11:16:57.941494 master-1 kubenswrapper[4771]: I1011 11:16:57.941350 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="fa11837b-6428-471c-b8d4-ffafd00d954b" containerName="reboot-os-networker-deploy-networkers" Oct 11 11:16:57.941494 master-1 kubenswrapper[4771]: I1011 11:16:57.941400 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="c5e880ca-1acb-4f3a-8d7c-98fa3c84e2ff" containerName="reboot-os-dataplane-edpm" Oct 11 11:16:57.942994 master-1 kubenswrapper[4771]: I1011 11:16:57.942950 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:57.948688 master-1 kubenswrapper[4771]: I1011 11:16:57.948600 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"networkers-neutron-metadata-default-certs-0" Oct 11 11:16:57.949448 master-1 kubenswrapper[4771]: I1011 11:16:57.949408 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"networkers-ovn-default-certs-0" Oct 11 11:16:57.949546 master-1 kubenswrapper[4771]: I1011 11:16:57.949491 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:16:57.949589 master-1 kubenswrapper[4771]: I1011 11:16:57.949535 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:16:57.949691 master-1 kubenswrapper[4771]: I1011 11:16:57.949653 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-networkers" Oct 11 11:16:57.968940 master-1 kubenswrapper[4771]: I1011 11:16:57.968882 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/install-certs-networker-deploy-networkers-nv56q"] Oct 11 11:16:58.034812 master-1 kubenswrapper[4771]: I1011 11:16:58.034723 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-inventory\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.035132 master-1 kubenswrapper[4771]: I1011 11:16:58.035042 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networkers-ovn-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-ovn-default-certs-0\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.035275 master-1 kubenswrapper[4771]: I1011 11:16:58.035236 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zgn\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-kube-api-access-d9zgn\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.035319 master-1 kubenswrapper[4771]: I1011 11:16:58.035297 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ssh-key\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.035577 master-1 kubenswrapper[4771]: I1011 11:16:58.035543 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-neutron-metadata-combined-ca-bundle\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.035722 master-1 kubenswrapper[4771]: I1011 11:16:58.035690 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networkers-neutron-metadata-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-neutron-metadata-default-certs-0\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.036215 master-1 kubenswrapper[4771]: I1011 11:16:58.036185 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-bootstrap-combined-ca-bundle\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.036279 master-1 kubenswrapper[4771]: I1011 11:16:58.036234 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ovn-combined-ca-bundle\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.138552 master-1 kubenswrapper[4771]: I1011 11:16:58.138446 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-inventory\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.138818 master-1 kubenswrapper[4771]: I1011 11:16:58.138762 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"networkers-ovn-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-ovn-default-certs-0\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.138926 master-1 kubenswrapper[4771]: I1011 11:16:58.138861 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zgn\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-kube-api-access-d9zgn\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.138985 master-1 kubenswrapper[4771]: I1011 11:16:58.138951 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ssh-key\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.139809 master-1 kubenswrapper[4771]: I1011 11:16:58.139740 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-neutron-metadata-combined-ca-bundle\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.139990 master-1 kubenswrapper[4771]: I1011 11:16:58.139944 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"networkers-neutron-metadata-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-neutron-metadata-default-certs-0\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.140157 master-1 kubenswrapper[4771]: I1011 11:16:58.140114 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-bootstrap-combined-ca-bundle\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.140250 master-1 kubenswrapper[4771]: I1011 11:16:58.140216 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ovn-combined-ca-bundle\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.145267 master-1 kubenswrapper[4771]: I1011 11:16:58.145206 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-neutron-metadata-combined-ca-bundle\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.145536 master-1 kubenswrapper[4771]: I1011 11:16:58.145474 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-inventory\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.146205 master-1 kubenswrapper[4771]: I1011 11:16:58.146145 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ovn-combined-ca-bundle\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.146278 master-1 kubenswrapper[4771]: I1011 11:16:58.146219 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-bootstrap-combined-ca-bundle\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.147274 master-1 kubenswrapper[4771]: I1011 11:16:58.147212 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"networkers-neutron-metadata-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-neutron-metadata-default-certs-0\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.149126 master-1 kubenswrapper[4771]: I1011 11:16:58.149053 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"networkers-ovn-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-ovn-default-certs-0\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.149720 master-1 kubenswrapper[4771]: I1011 11:16:58.149676 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ssh-key\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.165150 master-1 kubenswrapper[4771]: I1011 11:16:58.165068 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zgn\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-kube-api-access-d9zgn\") pod \"install-certs-networker-deploy-networkers-nv56q\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.277586 master-1 kubenswrapper[4771]: I1011 11:16:58.277343 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:16:58.870056 master-1 kubenswrapper[4771]: I1011 11:16:58.869955 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/install-certs-networker-deploy-networkers-nv56q"] Oct 11 11:16:58.877213 master-1 kubenswrapper[4771]: W1011 11:16:58.876890 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e295eb4_9a6f_493c_a3a9_85508f5d6f3c.slice/crio-2037c8e340f6cf5a5c0f8d8031e67b6348b08e864cc1806517bcb81191359f52 WatchSource:0}: Error finding container 2037c8e340f6cf5a5c0f8d8031e67b6348b08e864cc1806517bcb81191359f52: Status 404 returned error can't find the container with id 2037c8e340f6cf5a5c0f8d8031e67b6348b08e864cc1806517bcb81191359f52 Oct 11 11:16:59.814225 master-1 kubenswrapper[4771]: I1011 11:16:59.814159 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/install-certs-networker-deploy-networkers-nv56q" event={"ID":"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c","Type":"ContainerStarted","Data":"bb403e0a7b754caf72ba310593e321557413f8e6fc2973bf2fa8b211d6a1c105"} Oct 11 11:16:59.814480 master-1 kubenswrapper[4771]: I1011 11:16:59.814240 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/install-certs-networker-deploy-networkers-nv56q" event={"ID":"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c","Type":"ContainerStarted","Data":"2037c8e340f6cf5a5c0f8d8031e67b6348b08e864cc1806517bcb81191359f52"} Oct 11 11:16:59.850030 master-1 kubenswrapper[4771]: I1011 11:16:59.849930 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/install-certs-networker-deploy-networkers-nv56q" podStartSLOduration=2.424244444 podStartE2EDuration="2.849908014s" podCreationTimestamp="2025-10-11 11:16:57 +0000 UTC" firstStartedPulling="2025-10-11 11:16:58.880835918 +0000 UTC m=+3050.855062379" lastFinishedPulling="2025-10-11 11:16:59.306499468 +0000 UTC m=+3051.280725949" observedRunningTime="2025-10-11 11:16:59.844646073 +0000 UTC m=+3051.818872524" watchObservedRunningTime="2025-10-11 11:16:59.849908014 +0000 UTC m=+3051.824134465" Oct 11 11:17:10.267273 master-1 kubenswrapper[4771]: I1011 11:17:10.267176 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-operators-kdf87"] Oct 11 11:17:10.269495 master-1 kubenswrapper[4771]: I1011 11:17:10.269440 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.288436 master-1 kubenswrapper[4771]: I1011 11:17:10.287534 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg9sj\" (UniqueName: \"kubernetes.io/projected/7d441233-bd29-43df-9709-b6619d4ee7cb-kube-api-access-qg9sj\") pod \"redhat-operators-kdf87\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.288436 master-1 kubenswrapper[4771]: I1011 11:17:10.287771 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-utilities\") pod \"redhat-operators-kdf87\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.288436 master-1 kubenswrapper[4771]: I1011 11:17:10.287856 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-catalog-content\") pod \"redhat-operators-kdf87\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.306592 master-1 kubenswrapper[4771]: I1011 11:17:10.305137 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-kdf87"] Oct 11 11:17:10.391118 master-1 kubenswrapper[4771]: I1011 11:17:10.391043 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-utilities\") pod \"redhat-operators-kdf87\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.391411 master-1 kubenswrapper[4771]: I1011 11:17:10.391168 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-catalog-content\") pod \"redhat-operators-kdf87\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.391411 master-1 kubenswrapper[4771]: I1011 11:17:10.391384 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-qg9sj\" (UniqueName: \"kubernetes.io/projected/7d441233-bd29-43df-9709-b6619d4ee7cb-kube-api-access-qg9sj\") pod \"redhat-operators-kdf87\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.392213 master-1 kubenswrapper[4771]: I1011 11:17:10.392137 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-catalog-content\") pod \"redhat-operators-kdf87\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.392269 master-1 kubenswrapper[4771]: I1011 11:17:10.392225 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-utilities\") pod \"redhat-operators-kdf87\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.414565 master-1 kubenswrapper[4771]: I1011 11:17:10.413912 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg9sj\" (UniqueName: \"kubernetes.io/projected/7d441233-bd29-43df-9709-b6619d4ee7cb-kube-api-access-qg9sj\") pod \"redhat-operators-kdf87\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:10.603345 master-1 kubenswrapper[4771]: I1011 11:17:10.603271 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:11.102936 master-1 kubenswrapper[4771]: W1011 11:17:11.102863 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d441233_bd29_43df_9709_b6619d4ee7cb.slice/crio-0709253ecf6b2429f21c40dc085b220b079f71dc73b4ce3af67c0718902f13d2 WatchSource:0}: Error finding container 0709253ecf6b2429f21c40dc085b220b079f71dc73b4ce3af67c0718902f13d2: Status 404 returned error can't find the container with id 0709253ecf6b2429f21c40dc085b220b079f71dc73b4ce3af67c0718902f13d2 Oct 11 11:17:11.104948 master-1 kubenswrapper[4771]: I1011 11:17:11.104904 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-kdf87"] Oct 11 11:17:11.936382 master-1 kubenswrapper[4771]: I1011 11:17:11.934482 4771 generic.go:334] "Generic (PLEG): container finished" podID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerID="3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d" exitCode=0 Oct 11 11:17:11.936382 master-1 kubenswrapper[4771]: I1011 11:17:11.934573 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-kdf87" event={"ID":"7d441233-bd29-43df-9709-b6619d4ee7cb","Type":"ContainerDied","Data":"3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d"} Oct 11 11:17:11.936382 master-1 kubenswrapper[4771]: I1011 11:17:11.934665 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-kdf87" event={"ID":"7d441233-bd29-43df-9709-b6619d4ee7cb","Type":"ContainerStarted","Data":"0709253ecf6b2429f21c40dc085b220b079f71dc73b4ce3af67c0718902f13d2"} Oct 11 11:17:12.944159 master-1 kubenswrapper[4771]: I1011 11:17:12.944004 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-kdf87" event={"ID":"7d441233-bd29-43df-9709-b6619d4ee7cb","Type":"ContainerStarted","Data":"4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252"} Oct 11 11:17:13.955005 master-1 kubenswrapper[4771]: I1011 11:17:13.954924 4771 generic.go:334] "Generic (PLEG): container finished" podID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerID="4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252" exitCode=0 Oct 11 11:17:13.955005 master-1 kubenswrapper[4771]: I1011 11:17:13.954989 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-kdf87" event={"ID":"7d441233-bd29-43df-9709-b6619d4ee7cb","Type":"ContainerDied","Data":"4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252"} Oct 11 11:17:14.969191 master-1 kubenswrapper[4771]: I1011 11:17:14.969064 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-kdf87" event={"ID":"7d441233-bd29-43df-9709-b6619d4ee7cb","Type":"ContainerStarted","Data":"b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86"} Oct 11 11:17:14.999940 master-1 kubenswrapper[4771]: I1011 11:17:14.999844 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-operators-kdf87" podStartSLOduration=2.544242682 podStartE2EDuration="4.999825715s" podCreationTimestamp="2025-10-11 11:17:10 +0000 UTC" firstStartedPulling="2025-10-11 11:17:11.936966523 +0000 UTC m=+3063.911192984" lastFinishedPulling="2025-10-11 11:17:14.392549536 +0000 UTC m=+3066.366776017" observedRunningTime="2025-10-11 11:17:14.994659706 +0000 UTC m=+3066.968886157" watchObservedRunningTime="2025-10-11 11:17:14.999825715 +0000 UTC m=+3066.974052156" Oct 11 11:17:20.604458 master-1 kubenswrapper[4771]: I1011 11:17:20.604317 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:20.605659 master-1 kubenswrapper[4771]: I1011 11:17:20.604479 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:20.683849 master-1 kubenswrapper[4771]: I1011 11:17:20.683791 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:21.129915 master-1 kubenswrapper[4771]: I1011 11:17:21.129839 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:21.221509 master-1 kubenswrapper[4771]: I1011 11:17:21.221397 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-kdf87"] Oct 11 11:17:23.079729 master-1 kubenswrapper[4771]: I1011 11:17:23.079599 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-operators-kdf87" podUID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerName="registry-server" containerID="cri-o://b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86" gracePeriod=2 Oct 11 11:17:23.629428 master-1 kubenswrapper[4771]: I1011 11:17:23.628782 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:23.725037 master-1 kubenswrapper[4771]: I1011 11:17:23.724943 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-utilities\") pod \"7d441233-bd29-43df-9709-b6619d4ee7cb\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " Oct 11 11:17:23.725391 master-1 kubenswrapper[4771]: I1011 11:17:23.725070 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-catalog-content\") pod \"7d441233-bd29-43df-9709-b6619d4ee7cb\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " Oct 11 11:17:23.725484 master-1 kubenswrapper[4771]: I1011 11:17:23.725434 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg9sj\" (UniqueName: \"kubernetes.io/projected/7d441233-bd29-43df-9709-b6619d4ee7cb-kube-api-access-qg9sj\") pod \"7d441233-bd29-43df-9709-b6619d4ee7cb\" (UID: \"7d441233-bd29-43df-9709-b6619d4ee7cb\") " Oct 11 11:17:23.726078 master-1 kubenswrapper[4771]: I1011 11:17:23.725997 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-utilities" (OuterVolumeSpecName: "utilities") pod "7d441233-bd29-43df-9709-b6619d4ee7cb" (UID: "7d441233-bd29-43df-9709-b6619d4ee7cb"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:17:23.726175 master-1 kubenswrapper[4771]: I1011 11:17:23.726161 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:23.730886 master-1 kubenswrapper[4771]: I1011 11:17:23.730824 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d441233-bd29-43df-9709-b6619d4ee7cb-kube-api-access-qg9sj" (OuterVolumeSpecName: "kube-api-access-qg9sj") pod "7d441233-bd29-43df-9709-b6619d4ee7cb" (UID: "7d441233-bd29-43df-9709-b6619d4ee7cb"). InnerVolumeSpecName "kube-api-access-qg9sj". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:17:23.829524 master-1 kubenswrapper[4771]: I1011 11:17:23.829453 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-qg9sj\" (UniqueName: \"kubernetes.io/projected/7d441233-bd29-43df-9709-b6619d4ee7cb-kube-api-access-qg9sj\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:23.852083 master-1 kubenswrapper[4771]: I1011 11:17:23.851986 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "7d441233-bd29-43df-9709-b6619d4ee7cb" (UID: "7d441233-bd29-43df-9709-b6619d4ee7cb"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:17:23.933535 master-1 kubenswrapper[4771]: I1011 11:17:23.933306 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/7d441233-bd29-43df-9709-b6619d4ee7cb-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:24.097119 master-1 kubenswrapper[4771]: I1011 11:17:24.096988 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/install-certs-networker-deploy-networkers-nv56q" event={"ID":"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c","Type":"ContainerDied","Data":"bb403e0a7b754caf72ba310593e321557413f8e6fc2973bf2fa8b211d6a1c105"} Oct 11 11:17:24.098128 master-1 kubenswrapper[4771]: I1011 11:17:24.097000 4771 generic.go:334] "Generic (PLEG): container finished" podID="5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" containerID="bb403e0a7b754caf72ba310593e321557413f8e6fc2973bf2fa8b211d6a1c105" exitCode=0 Oct 11 11:17:24.101413 master-1 kubenswrapper[4771]: I1011 11:17:24.101296 4771 generic.go:334] "Generic (PLEG): container finished" podID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerID="b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86" exitCode=0 Oct 11 11:17:24.101615 master-1 kubenswrapper[4771]: I1011 11:17:24.101406 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-kdf87" event={"ID":"7d441233-bd29-43df-9709-b6619d4ee7cb","Type":"ContainerDied","Data":"b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86"} Oct 11 11:17:24.101615 master-1 kubenswrapper[4771]: I1011 11:17:24.101481 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-kdf87" Oct 11 11:17:24.101615 master-1 kubenswrapper[4771]: I1011 11:17:24.101519 4771 scope.go:117] "RemoveContainer" containerID="b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86" Oct 11 11:17:24.101940 master-1 kubenswrapper[4771]: I1011 11:17:24.101494 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-kdf87" event={"ID":"7d441233-bd29-43df-9709-b6619d4ee7cb","Type":"ContainerDied","Data":"0709253ecf6b2429f21c40dc085b220b079f71dc73b4ce3af67c0718902f13d2"} Oct 11 11:17:24.137659 master-1 kubenswrapper[4771]: I1011 11:17:24.137584 4771 scope.go:117] "RemoveContainer" containerID="4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252" Oct 11 11:17:24.186436 master-1 kubenswrapper[4771]: I1011 11:17:24.186180 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-kdf87"] Oct 11 11:17:24.190665 master-1 kubenswrapper[4771]: I1011 11:17:24.190584 4771 scope.go:117] "RemoveContainer" containerID="3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d" Oct 11 11:17:24.196020 master-1 kubenswrapper[4771]: I1011 11:17:24.195958 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-operators-kdf87"] Oct 11 11:17:24.230344 master-1 kubenswrapper[4771]: I1011 11:17:24.230268 4771 scope.go:117] "RemoveContainer" containerID="b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86" Oct 11 11:17:24.231253 master-1 kubenswrapper[4771]: E1011 11:17:24.231161 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86\": container with ID starting with b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86 not found: ID does not exist" containerID="b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86" Oct 11 11:17:24.231527 master-1 kubenswrapper[4771]: I1011 11:17:24.231252 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86"} err="failed to get container status \"b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86\": rpc error: code = NotFound desc = could not find container \"b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86\": container with ID starting with b4cac195b13f3409e289432f51c828555dddc2f13e084d003d4348b6ab307f86 not found: ID does not exist" Oct 11 11:17:24.231527 master-1 kubenswrapper[4771]: I1011 11:17:24.231304 4771 scope.go:117] "RemoveContainer" containerID="4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252" Oct 11 11:17:24.232193 master-1 kubenswrapper[4771]: E1011 11:17:24.232125 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252\": container with ID starting with 4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252 not found: ID does not exist" containerID="4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252" Oct 11 11:17:24.232311 master-1 kubenswrapper[4771]: I1011 11:17:24.232203 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252"} err="failed to get container status \"4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252\": rpc error: code = NotFound desc = could not find container \"4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252\": container with ID starting with 4c90b12391971b398abffe3daf33d2e40d9a40f331ec28b65522dbd2c0e41252 not found: ID does not exist" Oct 11 11:17:24.232311 master-1 kubenswrapper[4771]: I1011 11:17:24.232252 4771 scope.go:117] "RemoveContainer" containerID="3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d" Oct 11 11:17:24.232861 master-1 kubenswrapper[4771]: E1011 11:17:24.232808 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d\": container with ID starting with 3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d not found: ID does not exist" containerID="3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d" Oct 11 11:17:24.232971 master-1 kubenswrapper[4771]: I1011 11:17:24.232862 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d"} err="failed to get container status \"3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d\": rpc error: code = NotFound desc = could not find container \"3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d\": container with ID starting with 3137974f7b7302e607a1ee856871307db4a8f7114a1270eb04dee32b9ddba29d not found: ID does not exist" Oct 11 11:17:24.448467 master-1 kubenswrapper[4771]: I1011 11:17:24.448275 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d441233-bd29-43df-9709-b6619d4ee7cb" path="/var/lib/kubelet/pods/7d441233-bd29-43df-9709-b6619d4ee7cb/volumes" Oct 11 11:17:25.721866 master-1 kubenswrapper[4771]: I1011 11:17:25.721779 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:17:25.882196 master-1 kubenswrapper[4771]: I1011 11:17:25.882066 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ovn-combined-ca-bundle\") pod \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " Oct 11 11:17:25.882617 master-1 kubenswrapper[4771]: I1011 11:17:25.882223 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9zgn\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-kube-api-access-d9zgn\") pod \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " Oct 11 11:17:25.882617 master-1 kubenswrapper[4771]: I1011 11:17:25.882541 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-neutron-metadata-combined-ca-bundle\") pod \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " Oct 11 11:17:25.882905 master-1 kubenswrapper[4771]: I1011 11:17:25.882848 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"networkers-ovn-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-ovn-default-certs-0\") pod \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " Oct 11 11:17:25.883016 master-1 kubenswrapper[4771]: I1011 11:17:25.882971 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-inventory\") pod \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " Oct 11 11:17:25.883263 master-1 kubenswrapper[4771]: I1011 11:17:25.883205 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"networkers-neutron-metadata-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-neutron-metadata-default-certs-0\") pod \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " Oct 11 11:17:25.883463 master-1 kubenswrapper[4771]: I1011 11:17:25.883393 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ssh-key\") pod \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " Oct 11 11:17:25.883790 master-1 kubenswrapper[4771]: I1011 11:17:25.883463 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-bootstrap-combined-ca-bundle\") pod \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\" (UID: \"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c\") " Oct 11 11:17:25.888059 master-1 kubenswrapper[4771]: I1011 11:17:25.887966 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-neutron-metadata-combined-ca-bundle" (OuterVolumeSpecName: "neutron-metadata-combined-ca-bundle") pod "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" (UID: "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c"). InnerVolumeSpecName "neutron-metadata-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:17:25.888208 master-1 kubenswrapper[4771]: I1011 11:17:25.887990 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-bootstrap-combined-ca-bundle" (OuterVolumeSpecName: "bootstrap-combined-ca-bundle") pod "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" (UID: "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c"). InnerVolumeSpecName "bootstrap-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:17:25.889041 master-1 kubenswrapper[4771]: I1011 11:17:25.888971 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-ovn-default-certs-0" (OuterVolumeSpecName: "networkers-ovn-default-certs-0") pod "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" (UID: "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c"). InnerVolumeSpecName "networkers-ovn-default-certs-0". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:17:25.890547 master-1 kubenswrapper[4771]: I1011 11:17:25.890484 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ovn-combined-ca-bundle" (OuterVolumeSpecName: "ovn-combined-ca-bundle") pod "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" (UID: "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c"). InnerVolumeSpecName "ovn-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:17:25.890682 master-1 kubenswrapper[4771]: I1011 11:17:25.890615 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-kube-api-access-d9zgn" (OuterVolumeSpecName: "kube-api-access-d9zgn") pod "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" (UID: "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c"). InnerVolumeSpecName "kube-api-access-d9zgn". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:17:25.890997 master-1 kubenswrapper[4771]: I1011 11:17:25.890921 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-neutron-metadata-default-certs-0" (OuterVolumeSpecName: "networkers-neutron-metadata-default-certs-0") pod "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" (UID: "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c"). InnerVolumeSpecName "networkers-neutron-metadata-default-certs-0". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:17:25.911231 master-1 kubenswrapper[4771]: I1011 11:17:25.911164 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-inventory" (OuterVolumeSpecName: "inventory") pod "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" (UID: "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:17:25.925058 master-1 kubenswrapper[4771]: I1011 11:17:25.924949 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" (UID: "5e295eb4-9a6f-493c-a3a9-85508f5d6f3c"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:17:25.989201 master-1 kubenswrapper[4771]: I1011 11:17:25.989080 4771 reconciler_common.go:293] "Volume detached for volume \"networkers-ovn-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-ovn-default-certs-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:25.989201 master-1 kubenswrapper[4771]: I1011 11:17:25.989191 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:25.989543 master-1 kubenswrapper[4771]: I1011 11:17:25.989214 4771 reconciler_common.go:293] "Volume detached for volume \"networkers-neutron-metadata-default-certs-0\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-networkers-neutron-metadata-default-certs-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:25.989543 master-1 kubenswrapper[4771]: I1011 11:17:25.989237 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:25.989543 master-1 kubenswrapper[4771]: I1011 11:17:25.989255 4771 reconciler_common.go:293] "Volume detached for volume \"bootstrap-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-bootstrap-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:25.989543 master-1 kubenswrapper[4771]: I1011 11:17:25.989272 4771 reconciler_common.go:293] "Volume detached for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-ovn-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:25.989543 master-1 kubenswrapper[4771]: I1011 11:17:25.989291 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-d9zgn\" (UniqueName: \"kubernetes.io/projected/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-kube-api-access-d9zgn\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:25.989543 master-1 kubenswrapper[4771]: I1011 11:17:25.989310 4771 reconciler_common.go:293] "Volume detached for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5e295eb4-9a6f-493c-a3a9-85508f5d6f3c-neutron-metadata-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:17:26.135866 master-1 kubenswrapper[4771]: I1011 11:17:26.135610 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/install-certs-networker-deploy-networkers-nv56q" event={"ID":"5e295eb4-9a6f-493c-a3a9-85508f5d6f3c","Type":"ContainerDied","Data":"2037c8e340f6cf5a5c0f8d8031e67b6348b08e864cc1806517bcb81191359f52"} Oct 11 11:17:26.135866 master-1 kubenswrapper[4771]: I1011 11:17:26.135689 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/install-certs-networker-deploy-networkers-nv56q" Oct 11 11:17:26.135866 master-1 kubenswrapper[4771]: I1011 11:17:26.135693 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2037c8e340f6cf5a5c0f8d8031e67b6348b08e864cc1806517bcb81191359f52" Oct 11 11:17:26.278596 master-1 kubenswrapper[4771]: I1011 11:17:26.278482 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/ovn-networker-deploy-networkers-t2hq7"] Oct 11 11:17:26.279134 master-1 kubenswrapper[4771]: E1011 11:17:26.279088 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerName="extract-utilities" Oct 11 11:17:26.279134 master-1 kubenswrapper[4771]: I1011 11:17:26.279124 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerName="extract-utilities" Oct 11 11:17:26.279313 master-1 kubenswrapper[4771]: E1011 11:17:26.279198 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" containerName="install-certs-networker-deploy-networkers" Oct 11 11:17:26.279313 master-1 kubenswrapper[4771]: I1011 11:17:26.279216 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" containerName="install-certs-networker-deploy-networkers" Oct 11 11:17:26.279313 master-1 kubenswrapper[4771]: E1011 11:17:26.279246 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerName="extract-content" Oct 11 11:17:26.279313 master-1 kubenswrapper[4771]: I1011 11:17:26.279259 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerName="extract-content" Oct 11 11:17:26.279313 master-1 kubenswrapper[4771]: E1011 11:17:26.279286 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerName="registry-server" Oct 11 11:17:26.279313 master-1 kubenswrapper[4771]: I1011 11:17:26.279299 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerName="registry-server" Oct 11 11:17:26.279918 master-1 kubenswrapper[4771]: I1011 11:17:26.279827 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="5e295eb4-9a6f-493c-a3a9-85508f5d6f3c" containerName="install-certs-networker-deploy-networkers" Oct 11 11:17:26.279918 master-1 kubenswrapper[4771]: I1011 11:17:26.279887 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="7d441233-bd29-43df-9709-b6619d4ee7cb" containerName="registry-server" Oct 11 11:17:26.281345 master-1 kubenswrapper[4771]: I1011 11:17:26.281290 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.285239 master-1 kubenswrapper[4771]: I1011 11:17:26.285144 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-networkers" Oct 11 11:17:26.285875 master-1 kubenswrapper[4771]: I1011 11:17:26.285822 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"ovncontroller-config" Oct 11 11:17:26.287635 master-1 kubenswrapper[4771]: I1011 11:17:26.286276 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:17:26.287635 master-1 kubenswrapper[4771]: I1011 11:17:26.286318 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:17:26.295664 master-1 kubenswrapper[4771]: I1011 11:17:26.295593 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-networker-deploy-networkers-t2hq7"] Oct 11 11:17:26.400556 master-1 kubenswrapper[4771]: I1011 11:17:26.400346 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7bwg\" (UniqueName: \"kubernetes.io/projected/e8ccb67e-b425-48ea-a221-5991d470f77e-kube-api-access-h7bwg\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.400556 master-1 kubenswrapper[4771]: I1011 11:17:26.400505 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovncontroller-config-0\" (UniqueName: \"kubernetes.io/configmap/e8ccb67e-b425-48ea-a221-5991d470f77e-ovncontroller-config-0\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.401126 master-1 kubenswrapper[4771]: I1011 11:17:26.400983 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ssh-key\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.402164 master-1 kubenswrapper[4771]: I1011 11:17:26.402092 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ovn-combined-ca-bundle\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.402459 master-1 kubenswrapper[4771]: I1011 11:17:26.402392 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-inventory\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.506944 master-1 kubenswrapper[4771]: I1011 11:17:26.506793 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ssh-key\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.507286 master-1 kubenswrapper[4771]: I1011 11:17:26.507177 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ovn-combined-ca-bundle\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.507286 master-1 kubenswrapper[4771]: I1011 11:17:26.507268 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-inventory\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.507480 master-1 kubenswrapper[4771]: I1011 11:17:26.507434 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-h7bwg\" (UniqueName: \"kubernetes.io/projected/e8ccb67e-b425-48ea-a221-5991d470f77e-kube-api-access-h7bwg\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.507575 master-1 kubenswrapper[4771]: I1011 11:17:26.507509 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ovncontroller-config-0\" (UniqueName: \"kubernetes.io/configmap/e8ccb67e-b425-48ea-a221-5991d470f77e-ovncontroller-config-0\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.510016 master-1 kubenswrapper[4771]: I1011 11:17:26.509926 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovncontroller-config-0\" (UniqueName: \"kubernetes.io/configmap/e8ccb67e-b425-48ea-a221-5991d470f77e-ovncontroller-config-0\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.514719 master-1 kubenswrapper[4771]: I1011 11:17:26.514638 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-inventory\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.517419 master-1 kubenswrapper[4771]: I1011 11:17:26.517307 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ovn-combined-ca-bundle\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.527506 master-1 kubenswrapper[4771]: I1011 11:17:26.525694 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ssh-key\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.548317 master-1 kubenswrapper[4771]: I1011 11:17:26.548210 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7bwg\" (UniqueName: \"kubernetes.io/projected/e8ccb67e-b425-48ea-a221-5991d470f77e-kube-api-access-h7bwg\") pod \"ovn-networker-deploy-networkers-t2hq7\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:26.610535 master-1 kubenswrapper[4771]: I1011 11:17:26.610446 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:17:27.256507 master-1 kubenswrapper[4771]: I1011 11:17:27.256427 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/ovn-networker-deploy-networkers-t2hq7"] Oct 11 11:17:27.270331 master-1 kubenswrapper[4771]: W1011 11:17:27.270251 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8ccb67e_b425_48ea_a221_5991d470f77e.slice/crio-c868682262a8cb25efda7afaad016dfa9d45f465eeb3b92c6f246cbcf4775a5f WatchSource:0}: Error finding container c868682262a8cb25efda7afaad016dfa9d45f465eeb3b92c6f246cbcf4775a5f: Status 404 returned error can't find the container with id c868682262a8cb25efda7afaad016dfa9d45f465eeb3b92c6f246cbcf4775a5f Oct 11 11:17:28.159844 master-1 kubenswrapper[4771]: I1011 11:17:28.159727 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-networker-deploy-networkers-t2hq7" event={"ID":"e8ccb67e-b425-48ea-a221-5991d470f77e","Type":"ContainerStarted","Data":"ec469c05a1e6c2dba56551c551b8625e8765aef989a9c4f27620156c2420a755"} Oct 11 11:17:28.159844 master-1 kubenswrapper[4771]: I1011 11:17:28.159837 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-networker-deploy-networkers-t2hq7" event={"ID":"e8ccb67e-b425-48ea-a221-5991d470f77e","Type":"ContainerStarted","Data":"c868682262a8cb25efda7afaad016dfa9d45f465eeb3b92c6f246cbcf4775a5f"} Oct 11 11:17:28.191781 master-1 kubenswrapper[4771]: I1011 11:17:28.191677 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/ovn-networker-deploy-networkers-t2hq7" podStartSLOduration=1.758368256 podStartE2EDuration="2.191653792s" podCreationTimestamp="2025-10-11 11:17:26 +0000 UTC" firstStartedPulling="2025-10-11 11:17:27.27391078 +0000 UTC m=+3079.248137231" lastFinishedPulling="2025-10-11 11:17:27.707196326 +0000 UTC m=+3079.681422767" observedRunningTime="2025-10-11 11:17:28.187890614 +0000 UTC m=+3080.162117095" watchObservedRunningTime="2025-10-11 11:17:28.191653792 +0000 UTC m=+3080.165880233" Oct 11 11:18:31.650568 master-1 kubenswrapper[4771]: I1011 11:18:31.650478 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/neutron-metadata-dataplane-edpm-btpl5"] Oct 11 11:18:31.652407 master-1 kubenswrapper[4771]: I1011 11:18:31.652318 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.655129 master-1 kubenswrapper[4771]: I1011 11:18:31.655066 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"neutron-ovn-metadata-agent-neutron-config" Oct 11 11:18:31.655299 master-1 kubenswrapper[4771]: I1011 11:18:31.655225 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-metadata-neutron-config" Oct 11 11:18:31.657249 master-1 kubenswrapper[4771]: I1011 11:18:31.657033 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:18:31.684646 master-1 kubenswrapper[4771]: I1011 11:18:31.682718 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/neutron-metadata-dataplane-edpm-btpl5"] Oct 11 11:18:31.722454 master-1 kubenswrapper[4771]: I1011 11:18:31.722385 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-inventory\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.722649 master-1 kubenswrapper[4771]: I1011 11:18:31.722545 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-metadata-combined-ca-bundle\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.722649 master-1 kubenswrapper[4771]: I1011 11:18:31.722584 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-ssh-key\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.722649 master-1 kubenswrapper[4771]: I1011 11:18:31.722640 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-nova-metadata-neutron-config-0\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.722757 master-1 kubenswrapper[4771]: I1011 11:18:31.722695 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzcx4\" (UniqueName: \"kubernetes.io/projected/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-kube-api-access-bzcx4\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.722824 master-1 kubenswrapper[4771]: I1011 11:18:31.722781 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-ovn-metadata-agent-neutron-config-0\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.824722 master-1 kubenswrapper[4771]: I1011 11:18:31.824669 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-inventory\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.824982 master-1 kubenswrapper[4771]: I1011 11:18:31.824754 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-metadata-combined-ca-bundle\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.824982 master-1 kubenswrapper[4771]: I1011 11:18:31.824782 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-ssh-key\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.824982 master-1 kubenswrapper[4771]: I1011 11:18:31.824816 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-nova-metadata-neutron-config-0\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.824982 master-1 kubenswrapper[4771]: I1011 11:18:31.824861 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-bzcx4\" (UniqueName: \"kubernetes.io/projected/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-kube-api-access-bzcx4\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.824982 master-1 kubenswrapper[4771]: I1011 11:18:31.824926 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-ovn-metadata-agent-neutron-config-0\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.829103 master-1 kubenswrapper[4771]: I1011 11:18:31.829051 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-metadata-combined-ca-bundle\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.830523 master-1 kubenswrapper[4771]: I1011 11:18:31.830452 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-nova-metadata-neutron-config-0\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.830747 master-1 kubenswrapper[4771]: I1011 11:18:31.830581 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-ssh-key\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.832186 master-1 kubenswrapper[4771]: I1011 11:18:31.832150 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-ovn-metadata-agent-neutron-config-0\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.835143 master-1 kubenswrapper[4771]: I1011 11:18:31.835112 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-inventory\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.851380 master-1 kubenswrapper[4771]: I1011 11:18:31.851313 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzcx4\" (UniqueName: \"kubernetes.io/projected/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-kube-api-access-bzcx4\") pod \"neutron-metadata-dataplane-edpm-btpl5\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:31.982788 master-1 kubenswrapper[4771]: I1011 11:18:31.982584 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:18:32.617628 master-1 kubenswrapper[4771]: I1011 11:18:32.617568 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/neutron-metadata-dataplane-edpm-btpl5"] Oct 11 11:18:32.624034 master-1 kubenswrapper[4771]: W1011 11:18:32.623871 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2abfc2a_4d79_4b42_ab00_c7ae196304f0.slice/crio-387c5dbd0c37c97f3a2383936a353b710430ae82ed847558aff32c3c182df4b4 WatchSource:0}: Error finding container 387c5dbd0c37c97f3a2383936a353b710430ae82ed847558aff32c3c182df4b4: Status 404 returned error can't find the container with id 387c5dbd0c37c97f3a2383936a353b710430ae82ed847558aff32c3c182df4b4 Oct 11 11:18:32.900185 master-1 kubenswrapper[4771]: I1011 11:18:32.900118 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-metadata-dataplane-edpm-btpl5" event={"ID":"e2abfc2a-4d79-4b42-ab00-c7ae196304f0","Type":"ContainerStarted","Data":"387c5dbd0c37c97f3a2383936a353b710430ae82ed847558aff32c3c182df4b4"} Oct 11 11:18:33.915428 master-1 kubenswrapper[4771]: I1011 11:18:33.915195 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-metadata-dataplane-edpm-btpl5" event={"ID":"e2abfc2a-4d79-4b42-ab00-c7ae196304f0","Type":"ContainerStarted","Data":"eb46d2fb69688ed3bc12562768989eccbd773169ee43961ad0896310902fea8e"} Oct 11 11:18:33.952815 master-1 kubenswrapper[4771]: I1011 11:18:33.952669 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/neutron-metadata-dataplane-edpm-btpl5" podStartSLOduration=2.433834096 podStartE2EDuration="2.952642327s" podCreationTimestamp="2025-10-11 11:18:31 +0000 UTC" firstStartedPulling="2025-10-11 11:18:32.629270189 +0000 UTC m=+3144.603496630" lastFinishedPulling="2025-10-11 11:18:33.14807838 +0000 UTC m=+3145.122304861" observedRunningTime="2025-10-11 11:18:33.942309071 +0000 UTC m=+3145.916535592" watchObservedRunningTime="2025-10-11 11:18:33.952642327 +0000 UTC m=+3145.926868798" Oct 11 11:18:55.149192 master-1 kubenswrapper[4771]: I1011 11:18:55.149117 4771 generic.go:334] "Generic (PLEG): container finished" podID="e8ccb67e-b425-48ea-a221-5991d470f77e" containerID="ec469c05a1e6c2dba56551c551b8625e8765aef989a9c4f27620156c2420a755" exitCode=0 Oct 11 11:18:55.149846 master-1 kubenswrapper[4771]: I1011 11:18:55.149175 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-networker-deploy-networkers-t2hq7" event={"ID":"e8ccb67e-b425-48ea-a221-5991d470f77e","Type":"ContainerDied","Data":"ec469c05a1e6c2dba56551c551b8625e8765aef989a9c4f27620156c2420a755"} Oct 11 11:18:56.748012 master-1 kubenswrapper[4771]: I1011 11:18:56.747775 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:18:56.852912 master-1 kubenswrapper[4771]: I1011 11:18:56.852845 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ovn-combined-ca-bundle\") pod \"e8ccb67e-b425-48ea-a221-5991d470f77e\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " Oct 11 11:18:56.853144 master-1 kubenswrapper[4771]: I1011 11:18:56.853022 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7bwg\" (UniqueName: \"kubernetes.io/projected/e8ccb67e-b425-48ea-a221-5991d470f77e-kube-api-access-h7bwg\") pod \"e8ccb67e-b425-48ea-a221-5991d470f77e\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " Oct 11 11:18:56.853208 master-1 kubenswrapper[4771]: I1011 11:18:56.853190 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ssh-key\") pod \"e8ccb67e-b425-48ea-a221-5991d470f77e\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " Oct 11 11:18:56.853411 master-1 kubenswrapper[4771]: I1011 11:18:56.853361 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-inventory\") pod \"e8ccb67e-b425-48ea-a221-5991d470f77e\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " Oct 11 11:18:56.853479 master-1 kubenswrapper[4771]: I1011 11:18:56.853461 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ovncontroller-config-0\" (UniqueName: \"kubernetes.io/configmap/e8ccb67e-b425-48ea-a221-5991d470f77e-ovncontroller-config-0\") pod \"e8ccb67e-b425-48ea-a221-5991d470f77e\" (UID: \"e8ccb67e-b425-48ea-a221-5991d470f77e\") " Oct 11 11:18:56.857187 master-1 kubenswrapper[4771]: I1011 11:18:56.857094 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ccb67e-b425-48ea-a221-5991d470f77e-kube-api-access-h7bwg" (OuterVolumeSpecName: "kube-api-access-h7bwg") pod "e8ccb67e-b425-48ea-a221-5991d470f77e" (UID: "e8ccb67e-b425-48ea-a221-5991d470f77e"). InnerVolumeSpecName "kube-api-access-h7bwg". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:18:56.857392 master-1 kubenswrapper[4771]: I1011 11:18:56.857324 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ovn-combined-ca-bundle" (OuterVolumeSpecName: "ovn-combined-ca-bundle") pod "e8ccb67e-b425-48ea-a221-5991d470f77e" (UID: "e8ccb67e-b425-48ea-a221-5991d470f77e"). InnerVolumeSpecName "ovn-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:18:56.884528 master-1 kubenswrapper[4771]: I1011 11:18:56.884443 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ccb67e-b425-48ea-a221-5991d470f77e-ovncontroller-config-0" (OuterVolumeSpecName: "ovncontroller-config-0") pod "e8ccb67e-b425-48ea-a221-5991d470f77e" (UID: "e8ccb67e-b425-48ea-a221-5991d470f77e"). InnerVolumeSpecName "ovncontroller-config-0". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:18:56.887327 master-1 kubenswrapper[4771]: I1011 11:18:56.886159 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-inventory" (OuterVolumeSpecName: "inventory") pod "e8ccb67e-b425-48ea-a221-5991d470f77e" (UID: "e8ccb67e-b425-48ea-a221-5991d470f77e"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:18:56.890602 master-1 kubenswrapper[4771]: I1011 11:18:56.890488 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "e8ccb67e-b425-48ea-a221-5991d470f77e" (UID: "e8ccb67e-b425-48ea-a221-5991d470f77e"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:18:56.956939 master-1 kubenswrapper[4771]: I1011 11:18:56.956869 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:18:56.956939 master-1 kubenswrapper[4771]: I1011 11:18:56.956923 4771 reconciler_common.go:293] "Volume detached for volume \"ovncontroller-config-0\" (UniqueName: \"kubernetes.io/configmap/e8ccb67e-b425-48ea-a221-5991d470f77e-ovncontroller-config-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:18:56.956939 master-1 kubenswrapper[4771]: I1011 11:18:56.956936 4771 reconciler_common.go:293] "Volume detached for volume \"ovn-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ovn-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:18:56.956939 master-1 kubenswrapper[4771]: I1011 11:18:56.956946 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-h7bwg\" (UniqueName: \"kubernetes.io/projected/e8ccb67e-b425-48ea-a221-5991d470f77e-kube-api-access-h7bwg\") on node \"master-1\" DevicePath \"\"" Oct 11 11:18:56.956939 master-1 kubenswrapper[4771]: I1011 11:18:56.956960 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e8ccb67e-b425-48ea-a221-5991d470f77e-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:18:57.182341 master-1 kubenswrapper[4771]: I1011 11:18:57.181316 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/ovn-networker-deploy-networkers-t2hq7" event={"ID":"e8ccb67e-b425-48ea-a221-5991d470f77e","Type":"ContainerDied","Data":"c868682262a8cb25efda7afaad016dfa9d45f465eeb3b92c6f246cbcf4775a5f"} Oct 11 11:18:57.182341 master-1 kubenswrapper[4771]: I1011 11:18:57.181394 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c868682262a8cb25efda7afaad016dfa9d45f465eeb3b92c6f246cbcf4775a5f" Oct 11 11:18:57.182341 master-1 kubenswrapper[4771]: I1011 11:18:57.181370 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/ovn-networker-deploy-networkers-t2hq7" Oct 11 11:18:57.311975 master-1 kubenswrapper[4771]: I1011 11:18:57.311896 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/neutron-metadata-networker-deploy-networkers-p85s9"] Oct 11 11:18:57.312332 master-1 kubenswrapper[4771]: E1011 11:18:57.312298 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e8ccb67e-b425-48ea-a221-5991d470f77e" containerName="ovn-networker-deploy-networkers" Oct 11 11:18:57.312332 master-1 kubenswrapper[4771]: I1011 11:18:57.312323 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ccb67e-b425-48ea-a221-5991d470f77e" containerName="ovn-networker-deploy-networkers" Oct 11 11:18:57.312999 master-1 kubenswrapper[4771]: I1011 11:18:57.312543 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e8ccb67e-b425-48ea-a221-5991d470f77e" containerName="ovn-networker-deploy-networkers" Oct 11 11:18:57.313350 master-1 kubenswrapper[4771]: I1011 11:18:57.313317 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.317628 master-1 kubenswrapper[4771]: I1011 11:18:57.317597 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-networkers" Oct 11 11:18:57.330470 master-1 kubenswrapper[4771]: I1011 11:18:57.330414 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/neutron-metadata-networker-deploy-networkers-p85s9"] Oct 11 11:18:57.477450 master-1 kubenswrapper[4771]: I1011 11:18:57.477241 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4sxx\" (UniqueName: \"kubernetes.io/projected/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-kube-api-access-n4sxx\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.477450 master-1 kubenswrapper[4771]: I1011 11:18:57.477405 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-metadata-combined-ca-bundle\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.477768 master-1 kubenswrapper[4771]: I1011 11:18:57.477458 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-ovn-metadata-agent-neutron-config-0\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.477768 master-1 kubenswrapper[4771]: I1011 11:18:57.477676 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-ssh-key\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.477868 master-1 kubenswrapper[4771]: I1011 11:18:57.477787 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-nova-metadata-neutron-config-0\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.477919 master-1 kubenswrapper[4771]: I1011 11:18:57.477867 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-inventory\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.580417 master-1 kubenswrapper[4771]: I1011 11:18:57.580235 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-n4sxx\" (UniqueName: \"kubernetes.io/projected/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-kube-api-access-n4sxx\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.580980 master-1 kubenswrapper[4771]: I1011 11:18:57.580933 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-metadata-combined-ca-bundle\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.580980 master-1 kubenswrapper[4771]: I1011 11:18:57.580989 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-ovn-metadata-agent-neutron-config-0\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.581603 master-1 kubenswrapper[4771]: I1011 11:18:57.581568 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-ssh-key\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.581749 master-1 kubenswrapper[4771]: I1011 11:18:57.581683 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-nova-metadata-neutron-config-0\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.581749 master-1 kubenswrapper[4771]: I1011 11:18:57.581714 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-inventory\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.585514 master-1 kubenswrapper[4771]: I1011 11:18:57.585453 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-nova-metadata-neutron-config-0\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.585514 master-1 kubenswrapper[4771]: I1011 11:18:57.585472 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-metadata-combined-ca-bundle\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.586201 master-1 kubenswrapper[4771]: I1011 11:18:57.586134 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-ssh-key\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.586307 master-1 kubenswrapper[4771]: I1011 11:18:57.586283 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-ovn-metadata-agent-neutron-config-0\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.588458 master-1 kubenswrapper[4771]: I1011 11:18:57.587642 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-inventory\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.601365 master-1 kubenswrapper[4771]: I1011 11:18:57.601316 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4sxx\" (UniqueName: \"kubernetes.io/projected/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-kube-api-access-n4sxx\") pod \"neutron-metadata-networker-deploy-networkers-p85s9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:57.630910 master-1 kubenswrapper[4771]: I1011 11:18:57.630833 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:18:58.213259 master-1 kubenswrapper[4771]: I1011 11:18:58.213202 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/neutron-metadata-networker-deploy-networkers-p85s9"] Oct 11 11:18:59.204114 master-1 kubenswrapper[4771]: I1011 11:18:59.204058 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" event={"ID":"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9","Type":"ContainerStarted","Data":"92717e94e2058eedb11e073abfbbd407a95049fc4e25548ae9458ea5df577b0d"} Oct 11 11:18:59.204114 master-1 kubenswrapper[4771]: I1011 11:18:59.204119 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" event={"ID":"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9","Type":"ContainerStarted","Data":"29bf8aee9b4a890043d1883e13d1e18024c66bd6e92a8b19f57d72fb24e4d8ab"} Oct 11 11:18:59.231686 master-1 kubenswrapper[4771]: I1011 11:18:59.231576 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" podStartSLOduration=1.672994216 podStartE2EDuration="2.231557037s" podCreationTimestamp="2025-10-11 11:18:57 +0000 UTC" firstStartedPulling="2025-10-11 11:18:58.232921943 +0000 UTC m=+3170.207148424" lastFinishedPulling="2025-10-11 11:18:58.791484784 +0000 UTC m=+3170.765711245" observedRunningTime="2025-10-11 11:18:59.228922401 +0000 UTC m=+3171.203148892" watchObservedRunningTime="2025-10-11 11:18:59.231557037 +0000 UTC m=+3171.205783488" Oct 11 11:19:35.595755 master-1 kubenswrapper[4771]: I1011 11:19:35.595698 4771 generic.go:334] "Generic (PLEG): container finished" podID="e2abfc2a-4d79-4b42-ab00-c7ae196304f0" containerID="eb46d2fb69688ed3bc12562768989eccbd773169ee43961ad0896310902fea8e" exitCode=0 Oct 11 11:19:35.595755 master-1 kubenswrapper[4771]: I1011 11:19:35.595766 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-metadata-dataplane-edpm-btpl5" event={"ID":"e2abfc2a-4d79-4b42-ab00-c7ae196304f0","Type":"ContainerDied","Data":"eb46d2fb69688ed3bc12562768989eccbd773169ee43961ad0896310902fea8e"} Oct 11 11:19:37.160073 master-1 kubenswrapper[4771]: I1011 11:19:37.159903 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:19:37.297085 master-1 kubenswrapper[4771]: I1011 11:19:37.297006 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-inventory\") pod \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " Oct 11 11:19:37.297411 master-1 kubenswrapper[4771]: I1011 11:19:37.297349 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-nova-metadata-neutron-config-0\") pod \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " Oct 11 11:19:37.297480 master-1 kubenswrapper[4771]: I1011 11:19:37.297431 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-ssh-key\") pod \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " Oct 11 11:19:37.297569 master-1 kubenswrapper[4771]: I1011 11:19:37.297510 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-metadata-combined-ca-bundle\") pod \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " Oct 11 11:19:37.297648 master-1 kubenswrapper[4771]: I1011 11:19:37.297595 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-ovn-metadata-agent-neutron-config-0\") pod \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " Oct 11 11:19:37.297648 master-1 kubenswrapper[4771]: I1011 11:19:37.297640 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzcx4\" (UniqueName: \"kubernetes.io/projected/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-kube-api-access-bzcx4\") pod \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\" (UID: \"e2abfc2a-4d79-4b42-ab00-c7ae196304f0\") " Oct 11 11:19:37.301153 master-1 kubenswrapper[4771]: I1011 11:19:37.301096 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-metadata-combined-ca-bundle" (OuterVolumeSpecName: "neutron-metadata-combined-ca-bundle") pod "e2abfc2a-4d79-4b42-ab00-c7ae196304f0" (UID: "e2abfc2a-4d79-4b42-ab00-c7ae196304f0"). InnerVolumeSpecName "neutron-metadata-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:19:37.302125 master-1 kubenswrapper[4771]: I1011 11:19:37.302051 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-kube-api-access-bzcx4" (OuterVolumeSpecName: "kube-api-access-bzcx4") pod "e2abfc2a-4d79-4b42-ab00-c7ae196304f0" (UID: "e2abfc2a-4d79-4b42-ab00-c7ae196304f0"). InnerVolumeSpecName "kube-api-access-bzcx4". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:19:37.323133 master-1 kubenswrapper[4771]: I1011 11:19:37.323072 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-ovn-metadata-agent-neutron-config-0" (OuterVolumeSpecName: "neutron-ovn-metadata-agent-neutron-config-0") pod "e2abfc2a-4d79-4b42-ab00-c7ae196304f0" (UID: "e2abfc2a-4d79-4b42-ab00-c7ae196304f0"). InnerVolumeSpecName "neutron-ovn-metadata-agent-neutron-config-0". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:19:37.328680 master-1 kubenswrapper[4771]: I1011 11:19:37.328633 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-inventory" (OuterVolumeSpecName: "inventory") pod "e2abfc2a-4d79-4b42-ab00-c7ae196304f0" (UID: "e2abfc2a-4d79-4b42-ab00-c7ae196304f0"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:19:37.339450 master-1 kubenswrapper[4771]: I1011 11:19:37.339341 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-nova-metadata-neutron-config-0" (OuterVolumeSpecName: "nova-metadata-neutron-config-0") pod "e2abfc2a-4d79-4b42-ab00-c7ae196304f0" (UID: "e2abfc2a-4d79-4b42-ab00-c7ae196304f0"). InnerVolumeSpecName "nova-metadata-neutron-config-0". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:19:37.352269 master-1 kubenswrapper[4771]: I1011 11:19:37.351271 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "e2abfc2a-4d79-4b42-ab00-c7ae196304f0" (UID: "e2abfc2a-4d79-4b42-ab00-c7ae196304f0"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:19:37.400549 master-1 kubenswrapper[4771]: I1011 11:19:37.400447 4771 reconciler_common.go:293] "Volume detached for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-metadata-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:19:37.400549 master-1 kubenswrapper[4771]: I1011 11:19:37.400534 4771 reconciler_common.go:293] "Volume detached for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-neutron-ovn-metadata-agent-neutron-config-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:19:37.400549 master-1 kubenswrapper[4771]: I1011 11:19:37.400554 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-bzcx4\" (UniqueName: \"kubernetes.io/projected/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-kube-api-access-bzcx4\") on node \"master-1\" DevicePath \"\"" Oct 11 11:19:37.400917 master-1 kubenswrapper[4771]: I1011 11:19:37.400571 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:19:37.400917 master-1 kubenswrapper[4771]: I1011 11:19:37.400584 4771 reconciler_common.go:293] "Volume detached for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-nova-metadata-neutron-config-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:19:37.400917 master-1 kubenswrapper[4771]: I1011 11:19:37.400598 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/e2abfc2a-4d79-4b42-ab00-c7ae196304f0-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:19:37.617016 master-1 kubenswrapper[4771]: I1011 11:19:37.616945 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-metadata-dataplane-edpm-btpl5" event={"ID":"e2abfc2a-4d79-4b42-ab00-c7ae196304f0","Type":"ContainerDied","Data":"387c5dbd0c37c97f3a2383936a353b710430ae82ed847558aff32c3c182df4b4"} Oct 11 11:19:37.617016 master-1 kubenswrapper[4771]: I1011 11:19:37.617015 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="387c5dbd0c37c97f3a2383936a353b710430ae82ed847558aff32c3c182df4b4" Oct 11 11:19:37.617397 master-1 kubenswrapper[4771]: I1011 11:19:37.617144 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-metadata-dataplane-edpm-btpl5" Oct 11 11:19:37.758716 master-1 kubenswrapper[4771]: I1011 11:19:37.758565 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/libvirt-dataplane-edpm-9sl9q"] Oct 11 11:19:37.759124 master-1 kubenswrapper[4771]: E1011 11:19:37.759013 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="e2abfc2a-4d79-4b42-ab00-c7ae196304f0" containerName="neutron-metadata-dataplane-edpm" Oct 11 11:19:37.759124 master-1 kubenswrapper[4771]: I1011 11:19:37.759028 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2abfc2a-4d79-4b42-ab00-c7ae196304f0" containerName="neutron-metadata-dataplane-edpm" Oct 11 11:19:37.759289 master-1 kubenswrapper[4771]: I1011 11:19:37.759237 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="e2abfc2a-4d79-4b42-ab00-c7ae196304f0" containerName="neutron-metadata-dataplane-edpm" Oct 11 11:19:37.760074 master-1 kubenswrapper[4771]: I1011 11:19:37.760055 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.763460 master-1 kubenswrapper[4771]: I1011 11:19:37.763408 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:19:37.766563 master-1 kubenswrapper[4771]: I1011 11:19:37.766522 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"libvirt-secret" Oct 11 11:19:37.782119 master-1 kubenswrapper[4771]: I1011 11:19:37.782041 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/libvirt-dataplane-edpm-9sl9q"] Oct 11 11:19:37.807112 master-1 kubenswrapper[4771]: I1011 11:19:37.807038 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-inventory\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.807334 master-1 kubenswrapper[4771]: I1011 11:19:37.807128 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j577p\" (UniqueName: \"kubernetes.io/projected/85772d7d-920d-478b-88f8-6b7f135c79f4-kube-api-access-j577p\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.807334 master-1 kubenswrapper[4771]: I1011 11:19:37.807221 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-ssh-key\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.807334 master-1 kubenswrapper[4771]: I1011 11:19:37.807264 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"libvirt-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-combined-ca-bundle\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.807334 master-1 kubenswrapper[4771]: I1011 11:19:37.807302 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"libvirt-secret-0\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-secret-0\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.908830 master-1 kubenswrapper[4771]: I1011 11:19:37.908778 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-ssh-key\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.909159 master-1 kubenswrapper[4771]: I1011 11:19:37.909143 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"libvirt-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-combined-ca-bundle\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.909264 master-1 kubenswrapper[4771]: I1011 11:19:37.909251 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"libvirt-secret-0\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-secret-0\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.909448 master-1 kubenswrapper[4771]: I1011 11:19:37.909433 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-inventory\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.909556 master-1 kubenswrapper[4771]: I1011 11:19:37.909543 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-j577p\" (UniqueName: \"kubernetes.io/projected/85772d7d-920d-478b-88f8-6b7f135c79f4-kube-api-access-j577p\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.913760 master-1 kubenswrapper[4771]: I1011 11:19:37.913720 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-ssh-key\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.914891 master-1 kubenswrapper[4771]: I1011 11:19:37.914847 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"libvirt-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-combined-ca-bundle\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.915103 master-1 kubenswrapper[4771]: I1011 11:19:37.915035 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"libvirt-secret-0\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-secret-0\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.916298 master-1 kubenswrapper[4771]: I1011 11:19:37.915864 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-inventory\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:37.937406 master-1 kubenswrapper[4771]: I1011 11:19:37.937317 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-j577p\" (UniqueName: \"kubernetes.io/projected/85772d7d-920d-478b-88f8-6b7f135c79f4-kube-api-access-j577p\") pod \"libvirt-dataplane-edpm-9sl9q\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:38.153684 master-1 kubenswrapper[4771]: I1011 11:19:38.153334 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:19:38.720552 master-1 kubenswrapper[4771]: I1011 11:19:38.720463 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/libvirt-dataplane-edpm-9sl9q"] Oct 11 11:19:38.726702 master-1 kubenswrapper[4771]: W1011 11:19:38.726663 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85772d7d_920d_478b_88f8_6b7f135c79f4.slice/crio-469581d1f2ba91444a34fa09ee246cb89f27d8a7c5197db8e53776ec31618f65 WatchSource:0}: Error finding container 469581d1f2ba91444a34fa09ee246cb89f27d8a7c5197db8e53776ec31618f65: Status 404 returned error can't find the container with id 469581d1f2ba91444a34fa09ee246cb89f27d8a7c5197db8e53776ec31618f65 Oct 11 11:19:39.643549 master-1 kubenswrapper[4771]: I1011 11:19:39.643460 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/libvirt-dataplane-edpm-9sl9q" event={"ID":"85772d7d-920d-478b-88f8-6b7f135c79f4","Type":"ContainerStarted","Data":"5be7a0054307e1243100b620013bd83d4151a9dcec02bed8d5e9fdb9ac6d5d35"} Oct 11 11:19:39.643549 master-1 kubenswrapper[4771]: I1011 11:19:39.643541 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/libvirt-dataplane-edpm-9sl9q" event={"ID":"85772d7d-920d-478b-88f8-6b7f135c79f4","Type":"ContainerStarted","Data":"469581d1f2ba91444a34fa09ee246cb89f27d8a7c5197db8e53776ec31618f65"} Oct 11 11:19:39.677136 master-1 kubenswrapper[4771]: I1011 11:19:39.677006 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/libvirt-dataplane-edpm-9sl9q" podStartSLOduration=2.169731144 podStartE2EDuration="2.676975833s" podCreationTimestamp="2025-10-11 11:19:37 +0000 UTC" firstStartedPulling="2025-10-11 11:19:38.728961182 +0000 UTC m=+3210.703187663" lastFinishedPulling="2025-10-11 11:19:39.236205901 +0000 UTC m=+3211.210432352" observedRunningTime="2025-10-11 11:19:39.670065105 +0000 UTC m=+3211.644291626" watchObservedRunningTime="2025-10-11 11:19:39.676975833 +0000 UTC m=+3211.651202314" Oct 11 11:20:14.033114 master-1 kubenswrapper[4771]: I1011 11:20:14.033039 4771 generic.go:334] "Generic (PLEG): container finished" podID="050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" containerID="92717e94e2058eedb11e073abfbbd407a95049fc4e25548ae9458ea5df577b0d" exitCode=0 Oct 11 11:20:14.033114 master-1 kubenswrapper[4771]: I1011 11:20:14.033070 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" event={"ID":"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9","Type":"ContainerDied","Data":"92717e94e2058eedb11e073abfbbd407a95049fc4e25548ae9458ea5df577b0d"} Oct 11 11:20:15.781612 master-1 kubenswrapper[4771]: I1011 11:20:15.781543 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:20:15.881071 master-1 kubenswrapper[4771]: I1011 11:20:15.880968 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-nova-metadata-neutron-config-0\") pod \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " Oct 11 11:20:15.881427 master-1 kubenswrapper[4771]: I1011 11:20:15.881154 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-ssh-key\") pod \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " Oct 11 11:20:15.881590 master-1 kubenswrapper[4771]: I1011 11:20:15.881534 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-inventory\") pod \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " Oct 11 11:20:15.881732 master-1 kubenswrapper[4771]: I1011 11:20:15.881686 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-ovn-metadata-agent-neutron-config-0\") pod \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " Oct 11 11:20:15.881820 master-1 kubenswrapper[4771]: I1011 11:20:15.881763 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4sxx\" (UniqueName: \"kubernetes.io/projected/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-kube-api-access-n4sxx\") pod \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " Oct 11 11:20:15.881820 master-1 kubenswrapper[4771]: I1011 11:20:15.881807 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-metadata-combined-ca-bundle\") pod \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\" (UID: \"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9\") " Oct 11 11:20:15.887927 master-1 kubenswrapper[4771]: I1011 11:20:15.887857 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-metadata-combined-ca-bundle" (OuterVolumeSpecName: "neutron-metadata-combined-ca-bundle") pod "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" (UID: "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9"). InnerVolumeSpecName "neutron-metadata-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:20:15.888489 master-1 kubenswrapper[4771]: I1011 11:20:15.888412 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-kube-api-access-n4sxx" (OuterVolumeSpecName: "kube-api-access-n4sxx") pod "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" (UID: "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9"). InnerVolumeSpecName "kube-api-access-n4sxx". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:20:15.917054 master-1 kubenswrapper[4771]: I1011 11:20:15.916965 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" (UID: "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:20:15.933189 master-1 kubenswrapper[4771]: I1011 11:20:15.933124 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-ovn-metadata-agent-neutron-config-0" (OuterVolumeSpecName: "neutron-ovn-metadata-agent-neutron-config-0") pod "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" (UID: "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9"). InnerVolumeSpecName "neutron-ovn-metadata-agent-neutron-config-0". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:20:15.934695 master-1 kubenswrapper[4771]: I1011 11:20:15.934604 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-nova-metadata-neutron-config-0" (OuterVolumeSpecName: "nova-metadata-neutron-config-0") pod "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" (UID: "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9"). InnerVolumeSpecName "nova-metadata-neutron-config-0". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:20:15.937053 master-1 kubenswrapper[4771]: I1011 11:20:15.936542 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-inventory" (OuterVolumeSpecName: "inventory") pod "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" (UID: "050ae5f1-9a55-4c0b-8b83-f353aff0b3a9"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:20:15.984220 master-1 kubenswrapper[4771]: I1011 11:20:15.984136 4771 reconciler_common.go:293] "Volume detached for volume \"neutron-ovn-metadata-agent-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-ovn-metadata-agent-neutron-config-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:20:15.984220 master-1 kubenswrapper[4771]: I1011 11:20:15.984194 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-n4sxx\" (UniqueName: \"kubernetes.io/projected/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-kube-api-access-n4sxx\") on node \"master-1\" DevicePath \"\"" Oct 11 11:20:15.984220 master-1 kubenswrapper[4771]: I1011 11:20:15.984211 4771 reconciler_common.go:293] "Volume detached for volume \"neutron-metadata-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-neutron-metadata-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:20:15.984220 master-1 kubenswrapper[4771]: I1011 11:20:15.984228 4771 reconciler_common.go:293] "Volume detached for volume \"nova-metadata-neutron-config-0\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-nova-metadata-neutron-config-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:20:15.984652 master-1 kubenswrapper[4771]: I1011 11:20:15.984242 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:20:15.984652 master-1 kubenswrapper[4771]: I1011 11:20:15.984256 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/050ae5f1-9a55-4c0b-8b83-f353aff0b3a9-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:20:16.061144 master-1 kubenswrapper[4771]: I1011 11:20:16.061066 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" event={"ID":"050ae5f1-9a55-4c0b-8b83-f353aff0b3a9","Type":"ContainerDied","Data":"29bf8aee9b4a890043d1883e13d1e18024c66bd6e92a8b19f57d72fb24e4d8ab"} Oct 11 11:20:16.061611 master-1 kubenswrapper[4771]: I1011 11:20:16.061590 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29bf8aee9b4a890043d1883e13d1e18024c66bd6e92a8b19f57d72fb24e4d8ab" Oct 11 11:20:16.062091 master-1 kubenswrapper[4771]: I1011 11:20:16.061188 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/neutron-metadata-networker-deploy-networkers-p85s9" Oct 11 11:21:34.115548 master-1 kubenswrapper[4771]: I1011 11:21:34.115417 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-marketplace-tvfgn"] Oct 11 11:21:34.116576 master-1 kubenswrapper[4771]: E1011 11:21:34.115801 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" containerName="neutron-metadata-networker-deploy-networkers" Oct 11 11:21:34.116576 master-1 kubenswrapper[4771]: I1011 11:21:34.115816 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" containerName="neutron-metadata-networker-deploy-networkers" Oct 11 11:21:34.116576 master-1 kubenswrapper[4771]: I1011 11:21:34.115964 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="050ae5f1-9a55-4c0b-8b83-f353aff0b3a9" containerName="neutron-metadata-networker-deploy-networkers" Oct 11 11:21:34.117388 master-1 kubenswrapper[4771]: I1011 11:21:34.117335 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.155270 master-1 kubenswrapper[4771]: I1011 11:21:34.155135 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-tvfgn"] Oct 11 11:21:34.204932 master-1 kubenswrapper[4771]: I1011 11:21:34.204872 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-catalog-content\") pod \"redhat-marketplace-tvfgn\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.205338 master-1 kubenswrapper[4771]: I1011 11:21:34.205013 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-utilities\") pod \"redhat-marketplace-tvfgn\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.205338 master-1 kubenswrapper[4771]: I1011 11:21:34.205100 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrd5q\" (UniqueName: \"kubernetes.io/projected/127d4d79-6da3-4be4-9271-fb04d6d3fb06-kube-api-access-jrd5q\") pod \"redhat-marketplace-tvfgn\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.307480 master-1 kubenswrapper[4771]: I1011 11:21:34.307382 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-catalog-content\") pod \"redhat-marketplace-tvfgn\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.307744 master-1 kubenswrapper[4771]: I1011 11:21:34.307559 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-utilities\") pod \"redhat-marketplace-tvfgn\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.307744 master-1 kubenswrapper[4771]: I1011 11:21:34.307685 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-jrd5q\" (UniqueName: \"kubernetes.io/projected/127d4d79-6da3-4be4-9271-fb04d6d3fb06-kube-api-access-jrd5q\") pod \"redhat-marketplace-tvfgn\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.308236 master-1 kubenswrapper[4771]: I1011 11:21:34.307954 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-catalog-content\") pod \"redhat-marketplace-tvfgn\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.308867 master-1 kubenswrapper[4771]: I1011 11:21:34.308395 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-utilities\") pod \"redhat-marketplace-tvfgn\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.336311 master-1 kubenswrapper[4771]: I1011 11:21:34.336240 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrd5q\" (UniqueName: \"kubernetes.io/projected/127d4d79-6da3-4be4-9271-fb04d6d3fb06-kube-api-access-jrd5q\") pod \"redhat-marketplace-tvfgn\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.447257 master-1 kubenswrapper[4771]: I1011 11:21:34.446645 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:34.941748 master-1 kubenswrapper[4771]: I1011 11:21:34.941686 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-tvfgn"] Oct 11 11:21:35.950029 master-1 kubenswrapper[4771]: I1011 11:21:35.949954 4771 generic.go:334] "Generic (PLEG): container finished" podID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerID="1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142" exitCode=0 Oct 11 11:21:35.950796 master-1 kubenswrapper[4771]: I1011 11:21:35.950085 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-tvfgn" event={"ID":"127d4d79-6da3-4be4-9271-fb04d6d3fb06","Type":"ContainerDied","Data":"1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142"} Oct 11 11:21:35.950796 master-1 kubenswrapper[4771]: I1011 11:21:35.950152 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-tvfgn" event={"ID":"127d4d79-6da3-4be4-9271-fb04d6d3fb06","Type":"ContainerStarted","Data":"bb0de95a85571686983444043ed59014f3eebfb860364be2717ad7ec4716bf05"} Oct 11 11:21:35.953197 master-1 kubenswrapper[4771]: I1011 11:21:35.953042 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 11:21:37.983507 master-1 kubenswrapper[4771]: I1011 11:21:37.983423 4771 generic.go:334] "Generic (PLEG): container finished" podID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerID="80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09" exitCode=0 Oct 11 11:21:37.984525 master-1 kubenswrapper[4771]: I1011 11:21:37.983514 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-tvfgn" event={"ID":"127d4d79-6da3-4be4-9271-fb04d6d3fb06","Type":"ContainerDied","Data":"80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09"} Oct 11 11:21:38.998597 master-1 kubenswrapper[4771]: I1011 11:21:38.998486 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-tvfgn" event={"ID":"127d4d79-6da3-4be4-9271-fb04d6d3fb06","Type":"ContainerStarted","Data":"db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e"} Oct 11 11:21:39.040416 master-1 kubenswrapper[4771]: I1011 11:21:39.040227 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-marketplace-tvfgn" podStartSLOduration=2.5833395919999997 podStartE2EDuration="5.040194241s" podCreationTimestamp="2025-10-11 11:21:34 +0000 UTC" firstStartedPulling="2025-10-11 11:21:35.952927383 +0000 UTC m=+3327.927153824" lastFinishedPulling="2025-10-11 11:21:38.409782032 +0000 UTC m=+3330.384008473" observedRunningTime="2025-10-11 11:21:39.034613631 +0000 UTC m=+3331.008840072" watchObservedRunningTime="2025-10-11 11:21:39.040194241 +0000 UTC m=+3331.014420682" Oct 11 11:21:44.450428 master-1 kubenswrapper[4771]: I1011 11:21:44.450131 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:44.450428 master-1 kubenswrapper[4771]: I1011 11:21:44.450287 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:44.503093 master-1 kubenswrapper[4771]: I1011 11:21:44.503025 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:45.114200 master-1 kubenswrapper[4771]: I1011 11:21:45.114130 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:45.194182 master-1 kubenswrapper[4771]: I1011 11:21:45.194109 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-tvfgn"] Oct 11 11:21:47.083758 master-1 kubenswrapper[4771]: I1011 11:21:47.083674 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-marketplace-tvfgn" podUID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerName="registry-server" containerID="cri-o://db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e" gracePeriod=2 Oct 11 11:21:47.691930 master-1 kubenswrapper[4771]: I1011 11:21:47.691861 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:47.754461 master-1 kubenswrapper[4771]: I1011 11:21:47.754308 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-utilities\") pod \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " Oct 11 11:21:47.754461 master-1 kubenswrapper[4771]: I1011 11:21:47.754466 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-catalog-content\") pod \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " Oct 11 11:21:47.754831 master-1 kubenswrapper[4771]: I1011 11:21:47.754558 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrd5q\" (UniqueName: \"kubernetes.io/projected/127d4d79-6da3-4be4-9271-fb04d6d3fb06-kube-api-access-jrd5q\") pod \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\" (UID: \"127d4d79-6da3-4be4-9271-fb04d6d3fb06\") " Oct 11 11:21:47.756082 master-1 kubenswrapper[4771]: I1011 11:21:47.756002 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-utilities" (OuterVolumeSpecName: "utilities") pod "127d4d79-6da3-4be4-9271-fb04d6d3fb06" (UID: "127d4d79-6da3-4be4-9271-fb04d6d3fb06"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:21:47.757889 master-1 kubenswrapper[4771]: I1011 11:21:47.757829 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127d4d79-6da3-4be4-9271-fb04d6d3fb06-kube-api-access-jrd5q" (OuterVolumeSpecName: "kube-api-access-jrd5q") pod "127d4d79-6da3-4be4-9271-fb04d6d3fb06" (UID: "127d4d79-6da3-4be4-9271-fb04d6d3fb06"). InnerVolumeSpecName "kube-api-access-jrd5q". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:21:47.778803 master-1 kubenswrapper[4771]: I1011 11:21:47.778671 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "127d4d79-6da3-4be4-9271-fb04d6d3fb06" (UID: "127d4d79-6da3-4be4-9271-fb04d6d3fb06"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:21:47.858131 master-1 kubenswrapper[4771]: I1011 11:21:47.857982 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:21:47.858131 master-1 kubenswrapper[4771]: I1011 11:21:47.858051 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/127d4d79-6da3-4be4-9271-fb04d6d3fb06-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:21:47.858131 master-1 kubenswrapper[4771]: I1011 11:21:47.858075 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-jrd5q\" (UniqueName: \"kubernetes.io/projected/127d4d79-6da3-4be4-9271-fb04d6d3fb06-kube-api-access-jrd5q\") on node \"master-1\" DevicePath \"\"" Oct 11 11:21:48.100759 master-1 kubenswrapper[4771]: I1011 11:21:48.100510 4771 generic.go:334] "Generic (PLEG): container finished" podID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerID="db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e" exitCode=0 Oct 11 11:21:48.100759 master-1 kubenswrapper[4771]: I1011 11:21:48.100610 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-tvfgn" event={"ID":"127d4d79-6da3-4be4-9271-fb04d6d3fb06","Type":"ContainerDied","Data":"db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e"} Oct 11 11:21:48.100759 master-1 kubenswrapper[4771]: I1011 11:21:48.100651 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-tvfgn" event={"ID":"127d4d79-6da3-4be4-9271-fb04d6d3fb06","Type":"ContainerDied","Data":"bb0de95a85571686983444043ed59014f3eebfb860364be2717ad7ec4716bf05"} Oct 11 11:21:48.100759 master-1 kubenswrapper[4771]: I1011 11:21:48.100693 4771 scope.go:117] "RemoveContainer" containerID="db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e" Oct 11 11:21:48.102107 master-1 kubenswrapper[4771]: I1011 11:21:48.100884 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-tvfgn" Oct 11 11:21:48.132104 master-1 kubenswrapper[4771]: I1011 11:21:48.131968 4771 scope.go:117] "RemoveContainer" containerID="80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09" Oct 11 11:21:48.166226 master-1 kubenswrapper[4771]: I1011 11:21:48.165401 4771 scope.go:117] "RemoveContainer" containerID="1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142" Oct 11 11:21:48.173916 master-1 kubenswrapper[4771]: I1011 11:21:48.173819 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-tvfgn"] Oct 11 11:21:48.183082 master-1 kubenswrapper[4771]: I1011 11:21:48.183019 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-marketplace-tvfgn"] Oct 11 11:21:48.230943 master-1 kubenswrapper[4771]: I1011 11:21:48.230863 4771 scope.go:117] "RemoveContainer" containerID="db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e" Oct 11 11:21:48.231505 master-1 kubenswrapper[4771]: E1011 11:21:48.231464 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e\": container with ID starting with db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e not found: ID does not exist" containerID="db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e" Oct 11 11:21:48.232241 master-1 kubenswrapper[4771]: I1011 11:21:48.231514 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e"} err="failed to get container status \"db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e\": rpc error: code = NotFound desc = could not find container \"db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e\": container with ID starting with db17f449ae0fb02b4d0c2bab45419797ba50f59a59b6ac45e42a324754b1a94e not found: ID does not exist" Oct 11 11:21:48.232241 master-1 kubenswrapper[4771]: I1011 11:21:48.231550 4771 scope.go:117] "RemoveContainer" containerID="80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09" Oct 11 11:21:48.232241 master-1 kubenswrapper[4771]: E1011 11:21:48.231995 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09\": container with ID starting with 80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09 not found: ID does not exist" containerID="80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09" Oct 11 11:21:48.232241 master-1 kubenswrapper[4771]: I1011 11:21:48.232084 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09"} err="failed to get container status \"80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09\": rpc error: code = NotFound desc = could not find container \"80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09\": container with ID starting with 80a371a805e99adac53136620f611284a90e75bd54e82d550133c8126203cc09 not found: ID does not exist" Oct 11 11:21:48.232241 master-1 kubenswrapper[4771]: I1011 11:21:48.232172 4771 scope.go:117] "RemoveContainer" containerID="1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142" Oct 11 11:21:48.233058 master-1 kubenswrapper[4771]: E1011 11:21:48.232995 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142\": container with ID starting with 1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142 not found: ID does not exist" containerID="1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142" Oct 11 11:21:48.233239 master-1 kubenswrapper[4771]: I1011 11:21:48.233069 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142"} err="failed to get container status \"1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142\": rpc error: code = NotFound desc = could not find container \"1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142\": container with ID starting with 1e35b38f0ac33b45160c40a25839f70590318147b90ddd186179931705b30142 not found: ID does not exist" Oct 11 11:21:48.452635 master-1 kubenswrapper[4771]: I1011 11:21:48.452421 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" path="/var/lib/kubelet/pods/127d4d79-6da3-4be4-9271-fb04d6d3fb06/volumes" Oct 11 11:24:22.981757 master-1 kubenswrapper[4771]: I1011 11:24:22.981654 4771 generic.go:334] "Generic (PLEG): container finished" podID="85772d7d-920d-478b-88f8-6b7f135c79f4" containerID="5be7a0054307e1243100b620013bd83d4151a9dcec02bed8d5e9fdb9ac6d5d35" exitCode=0 Oct 11 11:24:22.981757 master-1 kubenswrapper[4771]: I1011 11:24:22.981741 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/libvirt-dataplane-edpm-9sl9q" event={"ID":"85772d7d-920d-478b-88f8-6b7f135c79f4","Type":"ContainerDied","Data":"5be7a0054307e1243100b620013bd83d4151a9dcec02bed8d5e9fdb9ac6d5d35"} Oct 11 11:24:24.623392 master-1 kubenswrapper[4771]: I1011 11:24:24.623249 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:24:24.764607 master-1 kubenswrapper[4771]: I1011 11:24:24.763817 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"libvirt-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-combined-ca-bundle\") pod \"85772d7d-920d-478b-88f8-6b7f135c79f4\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " Oct 11 11:24:24.764607 master-1 kubenswrapper[4771]: I1011 11:24:24.763996 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-ssh-key\") pod \"85772d7d-920d-478b-88f8-6b7f135c79f4\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " Oct 11 11:24:24.764607 master-1 kubenswrapper[4771]: I1011 11:24:24.764022 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"libvirt-secret-0\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-secret-0\") pod \"85772d7d-920d-478b-88f8-6b7f135c79f4\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " Oct 11 11:24:24.764607 master-1 kubenswrapper[4771]: I1011 11:24:24.764182 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j577p\" (UniqueName: \"kubernetes.io/projected/85772d7d-920d-478b-88f8-6b7f135c79f4-kube-api-access-j577p\") pod \"85772d7d-920d-478b-88f8-6b7f135c79f4\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " Oct 11 11:24:24.764607 master-1 kubenswrapper[4771]: I1011 11:24:24.764234 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-inventory\") pod \"85772d7d-920d-478b-88f8-6b7f135c79f4\" (UID: \"85772d7d-920d-478b-88f8-6b7f135c79f4\") " Oct 11 11:24:24.769772 master-1 kubenswrapper[4771]: I1011 11:24:24.769687 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-combined-ca-bundle" (OuterVolumeSpecName: "libvirt-combined-ca-bundle") pod "85772d7d-920d-478b-88f8-6b7f135c79f4" (UID: "85772d7d-920d-478b-88f8-6b7f135c79f4"). InnerVolumeSpecName "libvirt-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:24:24.770491 master-1 kubenswrapper[4771]: I1011 11:24:24.770436 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85772d7d-920d-478b-88f8-6b7f135c79f4-kube-api-access-j577p" (OuterVolumeSpecName: "kube-api-access-j577p") pod "85772d7d-920d-478b-88f8-6b7f135c79f4" (UID: "85772d7d-920d-478b-88f8-6b7f135c79f4"). InnerVolumeSpecName "kube-api-access-j577p". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:24:24.791387 master-1 kubenswrapper[4771]: I1011 11:24:24.791233 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-inventory" (OuterVolumeSpecName: "inventory") pod "85772d7d-920d-478b-88f8-6b7f135c79f4" (UID: "85772d7d-920d-478b-88f8-6b7f135c79f4"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:24:24.796185 master-1 kubenswrapper[4771]: I1011 11:24:24.796090 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "85772d7d-920d-478b-88f8-6b7f135c79f4" (UID: "85772d7d-920d-478b-88f8-6b7f135c79f4"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:24:24.799351 master-1 kubenswrapper[4771]: I1011 11:24:24.799241 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-secret-0" (OuterVolumeSpecName: "libvirt-secret-0") pod "85772d7d-920d-478b-88f8-6b7f135c79f4" (UID: "85772d7d-920d-478b-88f8-6b7f135c79f4"). InnerVolumeSpecName "libvirt-secret-0". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:24:24.867128 master-1 kubenswrapper[4771]: I1011 11:24:24.866988 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:24:24.867128 master-1 kubenswrapper[4771]: I1011 11:24:24.867052 4771 reconciler_common.go:293] "Volume detached for volume \"libvirt-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:24:24.867128 master-1 kubenswrapper[4771]: I1011 11:24:24.867085 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:24:24.867128 master-1 kubenswrapper[4771]: I1011 11:24:24.867104 4771 reconciler_common.go:293] "Volume detached for volume \"libvirt-secret-0\" (UniqueName: \"kubernetes.io/secret/85772d7d-920d-478b-88f8-6b7f135c79f4-libvirt-secret-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:24:24.867128 master-1 kubenswrapper[4771]: I1011 11:24:24.867121 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-j577p\" (UniqueName: \"kubernetes.io/projected/85772d7d-920d-478b-88f8-6b7f135c79f4-kube-api-access-j577p\") on node \"master-1\" DevicePath \"\"" Oct 11 11:24:25.010591 master-1 kubenswrapper[4771]: I1011 11:24:25.010466 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/libvirt-dataplane-edpm-9sl9q" event={"ID":"85772d7d-920d-478b-88f8-6b7f135c79f4","Type":"ContainerDied","Data":"469581d1f2ba91444a34fa09ee246cb89f27d8a7c5197db8e53776ec31618f65"} Oct 11 11:24:25.010591 master-1 kubenswrapper[4771]: I1011 11:24:25.010540 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="469581d1f2ba91444a34fa09ee246cb89f27d8a7c5197db8e53776ec31618f65" Oct 11 11:24:25.010591 master-1 kubenswrapper[4771]: I1011 11:24:25.010580 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/libvirt-dataplane-edpm-9sl9q" Oct 11 11:24:25.140554 master-1 kubenswrapper[4771]: I1011 11:24:25.140286 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/nova-dataplane-edpm-w7vwd"] Oct 11 11:24:25.140980 master-1 kubenswrapper[4771]: E1011 11:24:25.140856 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerName="registry-server" Oct 11 11:24:25.140980 master-1 kubenswrapper[4771]: I1011 11:24:25.140878 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerName="registry-server" Oct 11 11:24:25.140980 master-1 kubenswrapper[4771]: E1011 11:24:25.140920 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerName="extract-content" Oct 11 11:24:25.140980 master-1 kubenswrapper[4771]: I1011 11:24:25.140928 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerName="extract-content" Oct 11 11:24:25.140980 master-1 kubenswrapper[4771]: E1011 11:24:25.140952 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerName="extract-utilities" Oct 11 11:24:25.140980 master-1 kubenswrapper[4771]: I1011 11:24:25.140959 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerName="extract-utilities" Oct 11 11:24:25.140980 master-1 kubenswrapper[4771]: E1011 11:24:25.140976 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="85772d7d-920d-478b-88f8-6b7f135c79f4" containerName="libvirt-dataplane-edpm" Oct 11 11:24:25.140980 master-1 kubenswrapper[4771]: I1011 11:24:25.140985 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="85772d7d-920d-478b-88f8-6b7f135c79f4" containerName="libvirt-dataplane-edpm" Oct 11 11:24:25.141348 master-1 kubenswrapper[4771]: I1011 11:24:25.141203 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="85772d7d-920d-478b-88f8-6b7f135c79f4" containerName="libvirt-dataplane-edpm" Oct 11 11:24:25.141348 master-1 kubenswrapper[4771]: I1011 11:24:25.141246 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="127d4d79-6da3-4be4-9271-fb04d6d3fb06" containerName="registry-server" Oct 11 11:24:25.142266 master-1 kubenswrapper[4771]: I1011 11:24:25.142201 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.145958 master-1 kubenswrapper[4771]: I1011 11:24:25.145890 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-cell1-compute-config" Oct 11 11:24:25.146228 master-1 kubenswrapper[4771]: I1011 11:24:25.146197 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"nova-migration-ssh-key" Oct 11 11:24:25.146309 master-1 kubenswrapper[4771]: I1011 11:24:25.146219 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:24:25.146441 master-1 kubenswrapper[4771]: I1011 11:24:25.146410 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:24:25.148214 master-1 kubenswrapper[4771]: I1011 11:24:25.148175 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:24:25.172031 master-1 kubenswrapper[4771]: I1011 11:24:25.171849 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-dataplane-edpm-w7vwd"] Oct 11 11:24:25.275455 master-1 kubenswrapper[4771]: I1011 11:24:25.275346 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-cell1-compute-config-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-0\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.275712 master-1 kubenswrapper[4771]: I1011 11:24:25.275512 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-migration-ssh-key-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-1\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.275790 master-1 kubenswrapper[4771]: I1011 11:24:25.275708 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-cell1-compute-config-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-1\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.276014 master-1 kubenswrapper[4771]: I1011 11:24:25.275963 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-migration-ssh-key-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-0\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.276142 master-1 kubenswrapper[4771]: I1011 11:24:25.276108 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c9vd\" (UniqueName: \"kubernetes.io/projected/06076001-16e9-4f5c-91c5-cf4a70441a10-kube-api-access-5c9vd\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.276281 master-1 kubenswrapper[4771]: I1011 11:24:25.276238 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-inventory\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.276452 master-1 kubenswrapper[4771]: I1011 11:24:25.276399 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nova-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-combined-ca-bundle\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.276540 master-1 kubenswrapper[4771]: I1011 11:24:25.276464 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-ssh-key\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.378309 master-1 kubenswrapper[4771]: I1011 11:24:25.378192 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-inventory\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.378878 master-1 kubenswrapper[4771]: I1011 11:24:25.378401 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-combined-ca-bundle\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.378878 master-1 kubenswrapper[4771]: I1011 11:24:25.378437 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-ssh-key\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.378878 master-1 kubenswrapper[4771]: I1011 11:24:25.378486 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-cell1-compute-config-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-0\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.378878 master-1 kubenswrapper[4771]: I1011 11:24:25.378734 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-migration-ssh-key-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-1\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.378878 master-1 kubenswrapper[4771]: I1011 11:24:25.378802 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-cell1-compute-config-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-1\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.378878 master-1 kubenswrapper[4771]: I1011 11:24:25.378885 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"nova-migration-ssh-key-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-0\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.379319 master-1 kubenswrapper[4771]: I1011 11:24:25.378935 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-5c9vd\" (UniqueName: \"kubernetes.io/projected/06076001-16e9-4f5c-91c5-cf4a70441a10-kube-api-access-5c9vd\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.384083 master-1 kubenswrapper[4771]: I1011 11:24:25.384015 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-ssh-key\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.384351 master-1 kubenswrapper[4771]: I1011 11:24:25.384293 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-combined-ca-bundle\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.384558 master-1 kubenswrapper[4771]: I1011 11:24:25.384482 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-inventory\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.385080 master-1 kubenswrapper[4771]: I1011 11:24:25.385031 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-cell1-compute-config-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-1\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.387313 master-1 kubenswrapper[4771]: I1011 11:24:25.387226 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-migration-ssh-key-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-1\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.387588 master-1 kubenswrapper[4771]: I1011 11:24:25.387337 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-migration-ssh-key-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-0\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.389239 master-1 kubenswrapper[4771]: I1011 11:24:25.389165 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"nova-cell1-compute-config-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-0\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.404932 master-1 kubenswrapper[4771]: I1011 11:24:25.404867 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c9vd\" (UniqueName: \"kubernetes.io/projected/06076001-16e9-4f5c-91c5-cf4a70441a10-kube-api-access-5c9vd\") pod \"nova-dataplane-edpm-w7vwd\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:25.470388 master-1 kubenswrapper[4771]: I1011 11:24:25.470263 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:24:26.117304 master-1 kubenswrapper[4771]: I1011 11:24:26.117227 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/nova-dataplane-edpm-w7vwd"] Oct 11 11:24:26.122285 master-1 kubenswrapper[4771]: W1011 11:24:26.122218 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06076001_16e9_4f5c_91c5_cf4a70441a10.slice/crio-29ce132ef5120873821167a745fbe8c43a3904410b9c4ead60e6d296b2b0a14f WatchSource:0}: Error finding container 29ce132ef5120873821167a745fbe8c43a3904410b9c4ead60e6d296b2b0a14f: Status 404 returned error can't find the container with id 29ce132ef5120873821167a745fbe8c43a3904410b9c4ead60e6d296b2b0a14f Oct 11 11:24:27.035627 master-1 kubenswrapper[4771]: I1011 11:24:27.035548 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-dataplane-edpm-w7vwd" event={"ID":"06076001-16e9-4f5c-91c5-cf4a70441a10","Type":"ContainerStarted","Data":"6a4c70e45eeacacda98edd84f348a353424f270b816032a92a2760dc6b017867"} Oct 11 11:24:27.035627 master-1 kubenswrapper[4771]: I1011 11:24:27.035629 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-dataplane-edpm-w7vwd" event={"ID":"06076001-16e9-4f5c-91c5-cf4a70441a10","Type":"ContainerStarted","Data":"29ce132ef5120873821167a745fbe8c43a3904410b9c4ead60e6d296b2b0a14f"} Oct 11 11:24:27.070739 master-1 kubenswrapper[4771]: I1011 11:24:27.070389 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/nova-dataplane-edpm-w7vwd" podStartSLOduration=1.577823888 podStartE2EDuration="2.070336608s" podCreationTimestamp="2025-10-11 11:24:25 +0000 UTC" firstStartedPulling="2025-10-11 11:24:26.125792726 +0000 UTC m=+3498.100019207" lastFinishedPulling="2025-10-11 11:24:26.618305436 +0000 UTC m=+3498.592531927" observedRunningTime="2025-10-11 11:24:27.057233673 +0000 UTC m=+3499.031460144" watchObservedRunningTime="2025-10-11 11:24:27.070336608 +0000 UTC m=+3499.044563079" Oct 11 11:26:18.125219 master-1 kubenswrapper[4771]: I1011 11:26:18.125134 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/community-operators-w2wmc"] Oct 11 11:26:18.128246 master-1 kubenswrapper[4771]: I1011 11:26:18.128204 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.163511 master-1 kubenswrapper[4771]: I1011 11:26:18.162993 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-w2wmc"] Oct 11 11:26:18.282720 master-1 kubenswrapper[4771]: I1011 11:26:18.282641 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-utilities\") pod \"community-operators-w2wmc\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.282970 master-1 kubenswrapper[4771]: I1011 11:26:18.282780 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-catalog-content\") pod \"community-operators-w2wmc\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.282970 master-1 kubenswrapper[4771]: I1011 11:26:18.282804 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xblgx\" (UniqueName: \"kubernetes.io/projected/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-kube-api-access-xblgx\") pod \"community-operators-w2wmc\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.384918 master-1 kubenswrapper[4771]: I1011 11:26:18.384771 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-utilities\") pod \"community-operators-w2wmc\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.384918 master-1 kubenswrapper[4771]: I1011 11:26:18.384916 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-catalog-content\") pod \"community-operators-w2wmc\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.385160 master-1 kubenswrapper[4771]: I1011 11:26:18.384942 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-xblgx\" (UniqueName: \"kubernetes.io/projected/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-kube-api-access-xblgx\") pod \"community-operators-w2wmc\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.386318 master-1 kubenswrapper[4771]: I1011 11:26:18.385779 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-utilities\") pod \"community-operators-w2wmc\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.386318 master-1 kubenswrapper[4771]: I1011 11:26:18.385851 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-catalog-content\") pod \"community-operators-w2wmc\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.408379 master-1 kubenswrapper[4771]: I1011 11:26:18.408319 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-xblgx\" (UniqueName: \"kubernetes.io/projected/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-kube-api-access-xblgx\") pod \"community-operators-w2wmc\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.447489 master-1 kubenswrapper[4771]: I1011 11:26:18.447429 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:18.945112 master-1 kubenswrapper[4771]: I1011 11:26:18.945066 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/community-operators-w2wmc"] Oct 11 11:26:19.300790 master-1 kubenswrapper[4771]: I1011 11:26:19.300570 4771 generic.go:334] "Generic (PLEG): container finished" podID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerID="fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab" exitCode=0 Oct 11 11:26:19.300790 master-1 kubenswrapper[4771]: I1011 11:26:19.300662 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-w2wmc" event={"ID":"8814fe28-55c9-4c68-b4a9-eb28935ac4e6","Type":"ContainerDied","Data":"fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab"} Oct 11 11:26:19.300790 master-1 kubenswrapper[4771]: I1011 11:26:19.300765 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-w2wmc" event={"ID":"8814fe28-55c9-4c68-b4a9-eb28935ac4e6","Type":"ContainerStarted","Data":"4d4154e2ffc86d619d5ee16fedba8023b4cee821e2a1d036f9f04cd2a1ed8991"} Oct 11 11:26:20.317897 master-1 kubenswrapper[4771]: I1011 11:26:20.317825 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-w2wmc" event={"ID":"8814fe28-55c9-4c68-b4a9-eb28935ac4e6","Type":"ContainerStarted","Data":"2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9"} Oct 11 11:26:21.334278 master-1 kubenswrapper[4771]: I1011 11:26:21.334159 4771 generic.go:334] "Generic (PLEG): container finished" podID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerID="2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9" exitCode=0 Oct 11 11:26:21.334278 master-1 kubenswrapper[4771]: I1011 11:26:21.334251 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-w2wmc" event={"ID":"8814fe28-55c9-4c68-b4a9-eb28935ac4e6","Type":"ContainerDied","Data":"2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9"} Oct 11 11:26:22.351302 master-1 kubenswrapper[4771]: I1011 11:26:22.351228 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-w2wmc" event={"ID":"8814fe28-55c9-4c68-b4a9-eb28935ac4e6","Type":"ContainerStarted","Data":"b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d"} Oct 11 11:26:22.388735 master-1 kubenswrapper[4771]: I1011 11:26:22.388617 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/community-operators-w2wmc" podStartSLOduration=1.691812393 podStartE2EDuration="4.38858277s" podCreationTimestamp="2025-10-11 11:26:18 +0000 UTC" firstStartedPulling="2025-10-11 11:26:19.303108801 +0000 UTC m=+3611.277335282" lastFinishedPulling="2025-10-11 11:26:21.999879218 +0000 UTC m=+3613.974105659" observedRunningTime="2025-10-11 11:26:22.38612508 +0000 UTC m=+3614.360351581" watchObservedRunningTime="2025-10-11 11:26:22.38858277 +0000 UTC m=+3614.362809251" Oct 11 11:26:28.454687 master-1 kubenswrapper[4771]: I1011 11:26:28.454526 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:28.455662 master-1 kubenswrapper[4771]: I1011 11:26:28.455634 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:28.508718 master-1 kubenswrapper[4771]: I1011 11:26:28.508664 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:29.490727 master-1 kubenswrapper[4771]: I1011 11:26:29.490629 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:29.587304 master-1 kubenswrapper[4771]: I1011 11:26:29.587204 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-w2wmc"] Oct 11 11:26:31.452251 master-1 kubenswrapper[4771]: I1011 11:26:31.452137 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/community-operators-w2wmc" podUID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerName="registry-server" containerID="cri-o://b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d" gracePeriod=2 Oct 11 11:26:32.070792 master-1 kubenswrapper[4771]: I1011 11:26:32.070762 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:32.176903 master-1 kubenswrapper[4771]: I1011 11:26:32.176821 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-catalog-content\") pod \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " Oct 11 11:26:32.177141 master-1 kubenswrapper[4771]: I1011 11:26:32.176975 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xblgx\" (UniqueName: \"kubernetes.io/projected/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-kube-api-access-xblgx\") pod \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " Oct 11 11:26:32.177997 master-1 kubenswrapper[4771]: I1011 11:26:32.177899 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-utilities\") pod \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\" (UID: \"8814fe28-55c9-4c68-b4a9-eb28935ac4e6\") " Oct 11 11:26:32.179903 master-1 kubenswrapper[4771]: I1011 11:26:32.179824 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-utilities" (OuterVolumeSpecName: "utilities") pod "8814fe28-55c9-4c68-b4a9-eb28935ac4e6" (UID: "8814fe28-55c9-4c68-b4a9-eb28935ac4e6"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:26:32.181719 master-1 kubenswrapper[4771]: I1011 11:26:32.181662 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-kube-api-access-xblgx" (OuterVolumeSpecName: "kube-api-access-xblgx") pod "8814fe28-55c9-4c68-b4a9-eb28935ac4e6" (UID: "8814fe28-55c9-4c68-b4a9-eb28935ac4e6"). InnerVolumeSpecName "kube-api-access-xblgx". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:26:32.228398 master-1 kubenswrapper[4771]: I1011 11:26:32.228288 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "8814fe28-55c9-4c68-b4a9-eb28935ac4e6" (UID: "8814fe28-55c9-4c68-b4a9-eb28935ac4e6"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:26:32.281873 master-1 kubenswrapper[4771]: I1011 11:26:32.281808 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:26:32.281873 master-1 kubenswrapper[4771]: I1011 11:26:32.281850 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-xblgx\" (UniqueName: \"kubernetes.io/projected/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-kube-api-access-xblgx\") on node \"master-1\" DevicePath \"\"" Oct 11 11:26:32.281873 master-1 kubenswrapper[4771]: I1011 11:26:32.281867 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/8814fe28-55c9-4c68-b4a9-eb28935ac4e6-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:26:32.462879 master-1 kubenswrapper[4771]: I1011 11:26:32.462800 4771 generic.go:334] "Generic (PLEG): container finished" podID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerID="b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d" exitCode=0 Oct 11 11:26:32.462879 master-1 kubenswrapper[4771]: I1011 11:26:32.462879 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-w2wmc" event={"ID":"8814fe28-55c9-4c68-b4a9-eb28935ac4e6","Type":"ContainerDied","Data":"b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d"} Oct 11 11:26:32.464505 master-1 kubenswrapper[4771]: I1011 11:26:32.462920 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/community-operators-w2wmc" Oct 11 11:26:32.464505 master-1 kubenswrapper[4771]: I1011 11:26:32.462948 4771 scope.go:117] "RemoveContainer" containerID="b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d" Oct 11 11:26:32.464505 master-1 kubenswrapper[4771]: I1011 11:26:32.462926 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/community-operators-w2wmc" event={"ID":"8814fe28-55c9-4c68-b4a9-eb28935ac4e6","Type":"ContainerDied","Data":"4d4154e2ffc86d619d5ee16fedba8023b4cee821e2a1d036f9f04cd2a1ed8991"} Oct 11 11:26:32.490310 master-1 kubenswrapper[4771]: I1011 11:26:32.490212 4771 scope.go:117] "RemoveContainer" containerID="2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9" Oct 11 11:26:32.518690 master-1 kubenswrapper[4771]: I1011 11:26:32.518506 4771 scope.go:117] "RemoveContainer" containerID="fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab" Oct 11 11:26:32.521338 master-1 kubenswrapper[4771]: I1011 11:26:32.521228 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/community-operators-w2wmc"] Oct 11 11:26:32.534638 master-1 kubenswrapper[4771]: I1011 11:26:32.534562 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/community-operators-w2wmc"] Oct 11 11:26:32.582678 master-1 kubenswrapper[4771]: I1011 11:26:32.580851 4771 scope.go:117] "RemoveContainer" containerID="b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d" Oct 11 11:26:32.582678 master-1 kubenswrapper[4771]: E1011 11:26:32.581615 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d\": container with ID starting with b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d not found: ID does not exist" containerID="b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d" Oct 11 11:26:32.582678 master-1 kubenswrapper[4771]: I1011 11:26:32.581666 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d"} err="failed to get container status \"b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d\": rpc error: code = NotFound desc = could not find container \"b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d\": container with ID starting with b45ca70a1032969936a1590655655e760522fa0042fc4efd3da92629d1ae6d2d not found: ID does not exist" Oct 11 11:26:32.582678 master-1 kubenswrapper[4771]: I1011 11:26:32.581696 4771 scope.go:117] "RemoveContainer" containerID="2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9" Oct 11 11:26:32.582678 master-1 kubenswrapper[4771]: E1011 11:26:32.582459 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9\": container with ID starting with 2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9 not found: ID does not exist" containerID="2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9" Oct 11 11:26:32.582678 master-1 kubenswrapper[4771]: I1011 11:26:32.582516 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9"} err="failed to get container status \"2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9\": rpc error: code = NotFound desc = could not find container \"2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9\": container with ID starting with 2767272dcf855a5de6642a9b5c31317bbeb02d57ce1544951aabe1404080fda9 not found: ID does not exist" Oct 11 11:26:32.582678 master-1 kubenswrapper[4771]: I1011 11:26:32.582556 4771 scope.go:117] "RemoveContainer" containerID="fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab" Oct 11 11:26:32.583167 master-1 kubenswrapper[4771]: E1011 11:26:32.583039 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab\": container with ID starting with fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab not found: ID does not exist" containerID="fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab" Oct 11 11:26:32.583167 master-1 kubenswrapper[4771]: I1011 11:26:32.583066 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab"} err="failed to get container status \"fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab\": rpc error: code = NotFound desc = could not find container \"fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab\": container with ID starting with fde57b07ddd785eb1143c36abc3c8d980d901ef8b84dd027eddebee0a6b62eab not found: ID does not exist" Oct 11 11:26:34.452545 master-1 kubenswrapper[4771]: I1011 11:26:34.452442 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" path="/var/lib/kubelet/pods/8814fe28-55c9-4c68-b4a9-eb28935ac4e6/volumes" Oct 11 11:28:10.665117 master-1 kubenswrapper[4771]: I1011 11:28:10.665025 4771 generic.go:334] "Generic (PLEG): container finished" podID="06076001-16e9-4f5c-91c5-cf4a70441a10" containerID="6a4c70e45eeacacda98edd84f348a353424f270b816032a92a2760dc6b017867" exitCode=0 Oct 11 11:28:10.665117 master-1 kubenswrapper[4771]: I1011 11:28:10.665090 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-dataplane-edpm-w7vwd" event={"ID":"06076001-16e9-4f5c-91c5-cf4a70441a10","Type":"ContainerDied","Data":"6a4c70e45eeacacda98edd84f348a353424f270b816032a92a2760dc6b017867"} Oct 11 11:28:12.312501 master-1 kubenswrapper[4771]: I1011 11:28:12.312457 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:28:12.399045 master-1 kubenswrapper[4771]: I1011 11:28:12.398985 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"nova-cell1-compute-config-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-0\") pod \"06076001-16e9-4f5c-91c5-cf4a70441a10\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " Oct 11 11:28:12.399277 master-1 kubenswrapper[4771]: I1011 11:28:12.399133 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"nova-migration-ssh-key-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-0\") pod \"06076001-16e9-4f5c-91c5-cf4a70441a10\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " Oct 11 11:28:12.399277 master-1 kubenswrapper[4771]: I1011 11:28:12.399270 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c9vd\" (UniqueName: \"kubernetes.io/projected/06076001-16e9-4f5c-91c5-cf4a70441a10-kube-api-access-5c9vd\") pod \"06076001-16e9-4f5c-91c5-cf4a70441a10\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " Oct 11 11:28:12.399372 master-1 kubenswrapper[4771]: I1011 11:28:12.399341 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"nova-migration-ssh-key-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-1\") pod \"06076001-16e9-4f5c-91c5-cf4a70441a10\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " Oct 11 11:28:12.399425 master-1 kubenswrapper[4771]: I1011 11:28:12.399378 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-inventory\") pod \"06076001-16e9-4f5c-91c5-cf4a70441a10\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " Oct 11 11:28:12.399425 master-1 kubenswrapper[4771]: I1011 11:28:12.399398 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-ssh-key\") pod \"06076001-16e9-4f5c-91c5-cf4a70441a10\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " Oct 11 11:28:12.399499 master-1 kubenswrapper[4771]: I1011 11:28:12.399437 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"nova-cell1-compute-config-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-1\") pod \"06076001-16e9-4f5c-91c5-cf4a70441a10\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " Oct 11 11:28:12.399499 master-1 kubenswrapper[4771]: I1011 11:28:12.399464 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"nova-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-combined-ca-bundle\") pod \"06076001-16e9-4f5c-91c5-cf4a70441a10\" (UID: \"06076001-16e9-4f5c-91c5-cf4a70441a10\") " Oct 11 11:28:12.403179 master-1 kubenswrapper[4771]: I1011 11:28:12.403074 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06076001-16e9-4f5c-91c5-cf4a70441a10-kube-api-access-5c9vd" (OuterVolumeSpecName: "kube-api-access-5c9vd") pod "06076001-16e9-4f5c-91c5-cf4a70441a10" (UID: "06076001-16e9-4f5c-91c5-cf4a70441a10"). InnerVolumeSpecName "kube-api-access-5c9vd". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:28:12.405473 master-1 kubenswrapper[4771]: I1011 11:28:12.405435 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-combined-ca-bundle" (OuterVolumeSpecName: "nova-combined-ca-bundle") pod "06076001-16e9-4f5c-91c5-cf4a70441a10" (UID: "06076001-16e9-4f5c-91c5-cf4a70441a10"). InnerVolumeSpecName "nova-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:28:12.422606 master-1 kubenswrapper[4771]: I1011 11:28:12.422526 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-1" (OuterVolumeSpecName: "nova-cell1-compute-config-1") pod "06076001-16e9-4f5c-91c5-cf4a70441a10" (UID: "06076001-16e9-4f5c-91c5-cf4a70441a10"). InnerVolumeSpecName "nova-cell1-compute-config-1". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:28:12.426011 master-1 kubenswrapper[4771]: I1011 11:28:12.425920 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-1" (OuterVolumeSpecName: "nova-migration-ssh-key-1") pod "06076001-16e9-4f5c-91c5-cf4a70441a10" (UID: "06076001-16e9-4f5c-91c5-cf4a70441a10"). InnerVolumeSpecName "nova-migration-ssh-key-1". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:28:12.429100 master-1 kubenswrapper[4771]: I1011 11:28:12.429047 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-0" (OuterVolumeSpecName: "nova-migration-ssh-key-0") pod "06076001-16e9-4f5c-91c5-cf4a70441a10" (UID: "06076001-16e9-4f5c-91c5-cf4a70441a10"). InnerVolumeSpecName "nova-migration-ssh-key-0". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:28:12.447213 master-1 kubenswrapper[4771]: I1011 11:28:12.447149 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-inventory" (OuterVolumeSpecName: "inventory") pod "06076001-16e9-4f5c-91c5-cf4a70441a10" (UID: "06076001-16e9-4f5c-91c5-cf4a70441a10"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:28:12.447996 master-1 kubenswrapper[4771]: I1011 11:28:12.447842 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "06076001-16e9-4f5c-91c5-cf4a70441a10" (UID: "06076001-16e9-4f5c-91c5-cf4a70441a10"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:28:12.449075 master-1 kubenswrapper[4771]: I1011 11:28:12.449028 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-0" (OuterVolumeSpecName: "nova-cell1-compute-config-0") pod "06076001-16e9-4f5c-91c5-cf4a70441a10" (UID: "06076001-16e9-4f5c-91c5-cf4a70441a10"). InnerVolumeSpecName "nova-cell1-compute-config-0". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:28:12.504275 master-1 kubenswrapper[4771]: I1011 11:28:12.503280 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-5c9vd\" (UniqueName: \"kubernetes.io/projected/06076001-16e9-4f5c-91c5-cf4a70441a10-kube-api-access-5c9vd\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:12.504275 master-1 kubenswrapper[4771]: I1011 11:28:12.503346 4771 reconciler_common.go:293] "Volume detached for volume \"nova-migration-ssh-key-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-1\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:12.504275 master-1 kubenswrapper[4771]: I1011 11:28:12.503398 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:12.504275 master-1 kubenswrapper[4771]: I1011 11:28:12.503417 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:12.504275 master-1 kubenswrapper[4771]: I1011 11:28:12.503436 4771 reconciler_common.go:293] "Volume detached for volume \"nova-cell1-compute-config-1\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-1\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:12.504275 master-1 kubenswrapper[4771]: I1011 11:28:12.503456 4771 reconciler_common.go:293] "Volume detached for volume \"nova-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:12.504275 master-1 kubenswrapper[4771]: I1011 11:28:12.503482 4771 reconciler_common.go:293] "Volume detached for volume \"nova-cell1-compute-config-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-cell1-compute-config-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:12.504275 master-1 kubenswrapper[4771]: I1011 11:28:12.503500 4771 reconciler_common.go:293] "Volume detached for volume \"nova-migration-ssh-key-0\" (UniqueName: \"kubernetes.io/secret/06076001-16e9-4f5c-91c5-cf4a70441a10-nova-migration-ssh-key-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:12.687213 master-1 kubenswrapper[4771]: I1011 11:28:12.687113 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/nova-dataplane-edpm-w7vwd" event={"ID":"06076001-16e9-4f5c-91c5-cf4a70441a10","Type":"ContainerDied","Data":"29ce132ef5120873821167a745fbe8c43a3904410b9c4ead60e6d296b2b0a14f"} Oct 11 11:28:12.687213 master-1 kubenswrapper[4771]: I1011 11:28:12.687166 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ce132ef5120873821167a745fbe8c43a3904410b9c4ead60e6d296b2b0a14f" Oct 11 11:28:12.687656 master-1 kubenswrapper[4771]: I1011 11:28:12.687265 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/nova-dataplane-edpm-w7vwd" Oct 11 11:28:12.852506 master-1 kubenswrapper[4771]: I1011 11:28:12.852449 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openstack/telemetry-dataplane-edpm-rdjt7"] Oct 11 11:28:12.852911 master-1 kubenswrapper[4771]: E1011 11:28:12.852889 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerName="registry-server" Oct 11 11:28:12.852973 master-1 kubenswrapper[4771]: I1011 11:28:12.852913 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerName="registry-server" Oct 11 11:28:12.852973 master-1 kubenswrapper[4771]: E1011 11:28:12.852948 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerName="extract-content" Oct 11 11:28:12.852973 master-1 kubenswrapper[4771]: I1011 11:28:12.852957 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerName="extract-content" Oct 11 11:28:12.852973 master-1 kubenswrapper[4771]: E1011 11:28:12.852972 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerName="extract-utilities" Oct 11 11:28:12.853100 master-1 kubenswrapper[4771]: I1011 11:28:12.852982 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerName="extract-utilities" Oct 11 11:28:12.853100 master-1 kubenswrapper[4771]: E1011 11:28:12.852992 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="06076001-16e9-4f5c-91c5-cf4a70441a10" containerName="nova-dataplane-edpm" Oct 11 11:28:12.853100 master-1 kubenswrapper[4771]: I1011 11:28:12.853001 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="06076001-16e9-4f5c-91c5-cf4a70441a10" containerName="nova-dataplane-edpm" Oct 11 11:28:12.853218 master-1 kubenswrapper[4771]: I1011 11:28:12.853194 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="06076001-16e9-4f5c-91c5-cf4a70441a10" containerName="nova-dataplane-edpm" Oct 11 11:28:12.853265 master-1 kubenswrapper[4771]: I1011 11:28:12.853223 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="8814fe28-55c9-4c68-b4a9-eb28935ac4e6" containerName="registry-server" Oct 11 11:28:12.854334 master-1 kubenswrapper[4771]: I1011 11:28:12.854298 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:12.857772 master-1 kubenswrapper[4771]: I1011 11:28:12.857679 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplanenodeset-edpm" Oct 11 11:28:12.857772 master-1 kubenswrapper[4771]: I1011 11:28:12.857703 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openstack"/"openstack-aee-default-env" Oct 11 11:28:12.857927 master-1 kubenswrapper[4771]: I1011 11:28:12.857755 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"ceilometer-compute-config-data" Oct 11 11:28:12.858507 master-1 kubenswrapper[4771]: I1011 11:28:12.858478 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openstack"/"dataplane-ansible-ssh-private-key-secret" Oct 11 11:28:12.860752 master-1 kubenswrapper[4771]: I1011 11:28:12.860706 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/telemetry-dataplane-edpm-rdjt7"] Oct 11 11:28:12.911716 master-1 kubenswrapper[4771]: I1011 11:28:12.911599 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ceilometer-compute-config-data-2\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-2\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:12.911716 master-1 kubenswrapper[4771]: I1011 11:28:12.911712 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-telemetry-combined-ca-bundle\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:12.912031 master-1 kubenswrapper[4771]: I1011 11:28:12.911764 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ssh-key\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:12.912031 master-1 kubenswrapper[4771]: I1011 11:28:12.911803 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ceilometer-compute-config-data-0\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-0\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:12.912031 master-1 kubenswrapper[4771]: I1011 11:28:12.911825 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzw24\" (UniqueName: \"kubernetes.io/projected/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-kube-api-access-hzw24\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:12.912031 master-1 kubenswrapper[4771]: I1011 11:28:12.911874 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ceilometer-compute-config-data-1\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-1\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:12.912239 master-1 kubenswrapper[4771]: I1011 11:28:12.912103 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-inventory\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.014741 master-1 kubenswrapper[4771]: I1011 11:28:13.014653 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ceilometer-compute-config-data-2\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-2\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.014741 master-1 kubenswrapper[4771]: I1011 11:28:13.014719 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"telemetry-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-telemetry-combined-ca-bundle\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.014741 master-1 kubenswrapper[4771]: I1011 11:28:13.014745 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ssh-key\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.015215 master-1 kubenswrapper[4771]: I1011 11:28:13.014776 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ceilometer-compute-config-data-0\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-0\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.015215 master-1 kubenswrapper[4771]: I1011 11:28:13.014814 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-hzw24\" (UniqueName: \"kubernetes.io/projected/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-kube-api-access-hzw24\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.015215 master-1 kubenswrapper[4771]: I1011 11:28:13.014888 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"ceilometer-compute-config-data-1\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-1\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.015215 master-1 kubenswrapper[4771]: I1011 11:28:13.015048 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-inventory\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.019844 master-1 kubenswrapper[4771]: I1011 11:28:13.019786 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ceilometer-compute-config-data-0\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-0\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.020000 master-1 kubenswrapper[4771]: I1011 11:28:13.019868 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ssh-key\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.020312 master-1 kubenswrapper[4771]: I1011 11:28:13.020265 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-inventory\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.021306 master-1 kubenswrapper[4771]: I1011 11:28:13.021258 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ceilometer-compute-config-data-1\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-1\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.021616 master-1 kubenswrapper[4771]: I1011 11:28:13.021559 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"telemetry-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-telemetry-combined-ca-bundle\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.022816 master-1 kubenswrapper[4771]: I1011 11:28:13.022758 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"ceilometer-compute-config-data-2\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-2\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.050766 master-1 kubenswrapper[4771]: I1011 11:28:13.050655 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzw24\" (UniqueName: \"kubernetes.io/projected/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-kube-api-access-hzw24\") pod \"telemetry-dataplane-edpm-rdjt7\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.175013 master-1 kubenswrapper[4771]: I1011 11:28:13.174853 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:28:13.842056 master-1 kubenswrapper[4771]: W1011 11:28:13.841988 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb1d9f67_e3e9_4545_9e5d_07f2c213fe55.slice/crio-d621ccd58559916f2503ebdd3bbdd04a63949a337855cab279f0c0f2f83d42a6 WatchSource:0}: Error finding container d621ccd58559916f2503ebdd3bbdd04a63949a337855cab279f0c0f2f83d42a6: Status 404 returned error can't find the container with id d621ccd58559916f2503ebdd3bbdd04a63949a337855cab279f0c0f2f83d42a6 Oct 11 11:28:13.846909 master-1 kubenswrapper[4771]: I1011 11:28:13.846867 4771 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 11 11:28:13.847120 master-1 kubenswrapper[4771]: I1011 11:28:13.847071 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openstack/telemetry-dataplane-edpm-rdjt7"] Oct 11 11:28:14.715069 master-1 kubenswrapper[4771]: I1011 11:28:14.714985 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/telemetry-dataplane-edpm-rdjt7" event={"ID":"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55","Type":"ContainerStarted","Data":"71b6cbea99b537fb97df8cac56078c84bd8fb6f72db7edf36ac99e71415f9777"} Oct 11 11:28:14.715069 master-1 kubenswrapper[4771]: I1011 11:28:14.715073 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/telemetry-dataplane-edpm-rdjt7" event={"ID":"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55","Type":"ContainerStarted","Data":"d621ccd58559916f2503ebdd3bbdd04a63949a337855cab279f0c0f2f83d42a6"} Oct 11 11:28:14.750576 master-1 kubenswrapper[4771]: I1011 11:28:14.750472 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openstack/telemetry-dataplane-edpm-rdjt7" podStartSLOduration=2.269454172 podStartE2EDuration="2.750451518s" podCreationTimestamp="2025-10-11 11:28:12 +0000 UTC" firstStartedPulling="2025-10-11 11:28:13.846759166 +0000 UTC m=+3725.820985637" lastFinishedPulling="2025-10-11 11:28:14.327756532 +0000 UTC m=+3726.301982983" observedRunningTime="2025-10-11 11:28:14.744323782 +0000 UTC m=+3726.718550223" watchObservedRunningTime="2025-10-11 11:28:14.750451518 +0000 UTC m=+3726.724677979" Oct 11 11:28:34.144021 master-1 kubenswrapper[4771]: I1011 11:28:34.142746 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-operators-lhdxz"] Oct 11 11:28:34.146633 master-1 kubenswrapper[4771]: I1011 11:28:34.146487 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.178677 master-1 kubenswrapper[4771]: I1011 11:28:34.178454 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-lhdxz"] Oct 11 11:28:34.242147 master-1 kubenswrapper[4771]: I1011 11:28:34.241786 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-catalog-content\") pod \"redhat-operators-lhdxz\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.242147 master-1 kubenswrapper[4771]: I1011 11:28:34.241877 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbp6l\" (UniqueName: \"kubernetes.io/projected/99934513-1a05-40c6-8b59-cca04d84b0cd-kube-api-access-fbp6l\") pod \"redhat-operators-lhdxz\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.242147 master-1 kubenswrapper[4771]: I1011 11:28:34.241924 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-utilities\") pod \"redhat-operators-lhdxz\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.344618 master-1 kubenswrapper[4771]: I1011 11:28:34.344528 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-catalog-content\") pod \"redhat-operators-lhdxz\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.345029 master-1 kubenswrapper[4771]: I1011 11:28:34.344674 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp6l\" (UniqueName: \"kubernetes.io/projected/99934513-1a05-40c6-8b59-cca04d84b0cd-kube-api-access-fbp6l\") pod \"redhat-operators-lhdxz\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.345029 master-1 kubenswrapper[4771]: I1011 11:28:34.344748 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-utilities\") pod \"redhat-operators-lhdxz\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.345944 master-1 kubenswrapper[4771]: I1011 11:28:34.345866 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-catalog-content\") pod \"redhat-operators-lhdxz\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.346016 master-1 kubenswrapper[4771]: I1011 11:28:34.345927 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-utilities\") pod \"redhat-operators-lhdxz\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.371484 master-1 kubenswrapper[4771]: I1011 11:28:34.371402 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbp6l\" (UniqueName: \"kubernetes.io/projected/99934513-1a05-40c6-8b59-cca04d84b0cd-kube-api-access-fbp6l\") pod \"redhat-operators-lhdxz\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.490485 master-1 kubenswrapper[4771]: I1011 11:28:34.490292 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:34.995456 master-1 kubenswrapper[4771]: W1011 11:28:34.995369 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99934513_1a05_40c6_8b59_cca04d84b0cd.slice/crio-75d4598e2a9b6653177ee29e0428d0a916a1effcd05dcf3ff168f9729f0343d7 WatchSource:0}: Error finding container 75d4598e2a9b6653177ee29e0428d0a916a1effcd05dcf3ff168f9729f0343d7: Status 404 returned error can't find the container with id 75d4598e2a9b6653177ee29e0428d0a916a1effcd05dcf3ff168f9729f0343d7 Oct 11 11:28:34.997226 master-1 kubenswrapper[4771]: I1011 11:28:34.997182 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-operators-lhdxz"] Oct 11 11:28:35.968405 master-1 kubenswrapper[4771]: I1011 11:28:35.967859 4771 generic.go:334] "Generic (PLEG): container finished" podID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerID="ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd" exitCode=0 Oct 11 11:28:35.968405 master-1 kubenswrapper[4771]: I1011 11:28:35.967910 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-lhdxz" event={"ID":"99934513-1a05-40c6-8b59-cca04d84b0cd","Type":"ContainerDied","Data":"ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd"} Oct 11 11:28:35.968405 master-1 kubenswrapper[4771]: I1011 11:28:35.967938 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-lhdxz" event={"ID":"99934513-1a05-40c6-8b59-cca04d84b0cd","Type":"ContainerStarted","Data":"75d4598e2a9b6653177ee29e0428d0a916a1effcd05dcf3ff168f9729f0343d7"} Oct 11 11:28:37.996599 master-1 kubenswrapper[4771]: I1011 11:28:37.996461 4771 generic.go:334] "Generic (PLEG): container finished" podID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerID="7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8" exitCode=0 Oct 11 11:28:37.996599 master-1 kubenswrapper[4771]: I1011 11:28:37.996526 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-lhdxz" event={"ID":"99934513-1a05-40c6-8b59-cca04d84b0cd","Type":"ContainerDied","Data":"7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8"} Oct 11 11:28:39.010726 master-1 kubenswrapper[4771]: I1011 11:28:39.010615 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-lhdxz" event={"ID":"99934513-1a05-40c6-8b59-cca04d84b0cd","Type":"ContainerStarted","Data":"bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37"} Oct 11 11:28:39.048160 master-1 kubenswrapper[4771]: I1011 11:28:39.048009 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-operators-lhdxz" podStartSLOduration=2.5197304149999997 podStartE2EDuration="5.047990424s" podCreationTimestamp="2025-10-11 11:28:34 +0000 UTC" firstStartedPulling="2025-10-11 11:28:35.974025516 +0000 UTC m=+3747.948251967" lastFinishedPulling="2025-10-11 11:28:38.502285515 +0000 UTC m=+3750.476511976" observedRunningTime="2025-10-11 11:28:39.04191988 +0000 UTC m=+3751.016146351" watchObservedRunningTime="2025-10-11 11:28:39.047990424 +0000 UTC m=+3751.022216875" Oct 11 11:28:44.490598 master-1 kubenswrapper[4771]: I1011 11:28:44.490502 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:44.490598 master-1 kubenswrapper[4771]: I1011 11:28:44.490588 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:45.545688 master-1 kubenswrapper[4771]: I1011 11:28:45.545565 4771 prober.go:107] "Probe failed" probeType="Startup" pod="openshift-marketplace/redhat-operators-lhdxz" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerName="registry-server" probeResult="failure" output=< Oct 11 11:28:45.545688 master-1 kubenswrapper[4771]: timeout: failed to connect service ":50051" within 1s Oct 11 11:28:45.545688 master-1 kubenswrapper[4771]: > Oct 11 11:28:54.573952 master-1 kubenswrapper[4771]: I1011 11:28:54.573868 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:54.662191 master-1 kubenswrapper[4771]: I1011 11:28:54.662122 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:54.830520 master-1 kubenswrapper[4771]: I1011 11:28:54.830335 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-lhdxz"] Oct 11 11:28:56.194128 master-1 kubenswrapper[4771]: I1011 11:28:56.194021 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-operators-lhdxz" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerName="registry-server" containerID="cri-o://bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37" gracePeriod=2 Oct 11 11:28:56.777694 master-1 kubenswrapper[4771]: I1011 11:28:56.777584 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:56.831719 master-1 kubenswrapper[4771]: I1011 11:28:56.831638 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbp6l\" (UniqueName: \"kubernetes.io/projected/99934513-1a05-40c6-8b59-cca04d84b0cd-kube-api-access-fbp6l\") pod \"99934513-1a05-40c6-8b59-cca04d84b0cd\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " Oct 11 11:28:56.832111 master-1 kubenswrapper[4771]: I1011 11:28:56.831836 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-catalog-content\") pod \"99934513-1a05-40c6-8b59-cca04d84b0cd\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " Oct 11 11:28:56.832111 master-1 kubenswrapper[4771]: I1011 11:28:56.831890 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-utilities\") pod \"99934513-1a05-40c6-8b59-cca04d84b0cd\" (UID: \"99934513-1a05-40c6-8b59-cca04d84b0cd\") " Oct 11 11:28:56.834006 master-1 kubenswrapper[4771]: I1011 11:28:56.833922 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-utilities" (OuterVolumeSpecName: "utilities") pod "99934513-1a05-40c6-8b59-cca04d84b0cd" (UID: "99934513-1a05-40c6-8b59-cca04d84b0cd"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:28:56.834956 master-1 kubenswrapper[4771]: I1011 11:28:56.834891 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99934513-1a05-40c6-8b59-cca04d84b0cd-kube-api-access-fbp6l" (OuterVolumeSpecName: "kube-api-access-fbp6l") pod "99934513-1a05-40c6-8b59-cca04d84b0cd" (UID: "99934513-1a05-40c6-8b59-cca04d84b0cd"). InnerVolumeSpecName "kube-api-access-fbp6l". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:28:56.920176 master-1 kubenswrapper[4771]: I1011 11:28:56.920077 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "99934513-1a05-40c6-8b59-cca04d84b0cd" (UID: "99934513-1a05-40c6-8b59-cca04d84b0cd"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:28:56.934554 master-1 kubenswrapper[4771]: I1011 11:28:56.934386 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:56.934554 master-1 kubenswrapper[4771]: I1011 11:28:56.934414 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/99934513-1a05-40c6-8b59-cca04d84b0cd-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:56.934554 master-1 kubenswrapper[4771]: I1011 11:28:56.934427 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-fbp6l\" (UniqueName: \"kubernetes.io/projected/99934513-1a05-40c6-8b59-cca04d84b0cd-kube-api-access-fbp6l\") on node \"master-1\" DevicePath \"\"" Oct 11 11:28:57.212393 master-1 kubenswrapper[4771]: I1011 11:28:57.212133 4771 generic.go:334] "Generic (PLEG): container finished" podID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerID="bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37" exitCode=0 Oct 11 11:28:57.212393 master-1 kubenswrapper[4771]: I1011 11:28:57.212228 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-lhdxz" event={"ID":"99934513-1a05-40c6-8b59-cca04d84b0cd","Type":"ContainerDied","Data":"bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37"} Oct 11 11:28:57.212393 master-1 kubenswrapper[4771]: I1011 11:28:57.212282 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-operators-lhdxz" Oct 11 11:28:57.212393 master-1 kubenswrapper[4771]: I1011 11:28:57.212318 4771 scope.go:117] "RemoveContainer" containerID="bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37" Oct 11 11:28:57.213135 master-1 kubenswrapper[4771]: I1011 11:28:57.212296 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-operators-lhdxz" event={"ID":"99934513-1a05-40c6-8b59-cca04d84b0cd","Type":"ContainerDied","Data":"75d4598e2a9b6653177ee29e0428d0a916a1effcd05dcf3ff168f9729f0343d7"} Oct 11 11:28:57.249206 master-1 kubenswrapper[4771]: I1011 11:28:57.249141 4771 scope.go:117] "RemoveContainer" containerID="7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8" Oct 11 11:28:57.283607 master-1 kubenswrapper[4771]: I1011 11:28:57.283518 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-operators-lhdxz"] Oct 11 11:28:57.291488 master-1 kubenswrapper[4771]: I1011 11:28:57.291425 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-operators-lhdxz"] Oct 11 11:28:57.301744 master-1 kubenswrapper[4771]: I1011 11:28:57.301168 4771 scope.go:117] "RemoveContainer" containerID="ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd" Oct 11 11:28:57.328391 master-1 kubenswrapper[4771]: I1011 11:28:57.328317 4771 scope.go:117] "RemoveContainer" containerID="bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37" Oct 11 11:28:57.329548 master-1 kubenswrapper[4771]: E1011 11:28:57.329494 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37\": container with ID starting with bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37 not found: ID does not exist" containerID="bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37" Oct 11 11:28:57.329672 master-1 kubenswrapper[4771]: I1011 11:28:57.329543 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37"} err="failed to get container status \"bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37\": rpc error: code = NotFound desc = could not find container \"bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37\": container with ID starting with bd68e0827538badf8a204719e423d94d72ac284e9f6fe9041ce8ef18c8fcdc37 not found: ID does not exist" Oct 11 11:28:57.329672 master-1 kubenswrapper[4771]: I1011 11:28:57.329571 4771 scope.go:117] "RemoveContainer" containerID="7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8" Oct 11 11:28:57.329933 master-1 kubenswrapper[4771]: E1011 11:28:57.329885 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8\": container with ID starting with 7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8 not found: ID does not exist" containerID="7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8" Oct 11 11:28:57.329933 master-1 kubenswrapper[4771]: I1011 11:28:57.329916 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8"} err="failed to get container status \"7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8\": rpc error: code = NotFound desc = could not find container \"7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8\": container with ID starting with 7c84458d8bbe597bd2d841d515cd3513825a2c87d56edcca948b8ed78e5eb4f8 not found: ID does not exist" Oct 11 11:28:57.329933 master-1 kubenswrapper[4771]: I1011 11:28:57.329934 4771 scope.go:117] "RemoveContainer" containerID="ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd" Oct 11 11:28:57.330340 master-1 kubenswrapper[4771]: E1011 11:28:57.330268 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd\": container with ID starting with ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd not found: ID does not exist" containerID="ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd" Oct 11 11:28:57.330485 master-1 kubenswrapper[4771]: I1011 11:28:57.330342 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd"} err="failed to get container status \"ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd\": rpc error: code = NotFound desc = could not find container \"ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd\": container with ID starting with ce4d5eb6c9a45eb921d73906256a74d113968eab42b2553898b933519e5b9dfd not found: ID does not exist" Oct 11 11:28:58.449302 master-1 kubenswrapper[4771]: I1011 11:28:58.449218 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" path="/var/lib/kubelet/pods/99934513-1a05-40c6-8b59-cca04d84b0cd/volumes" Oct 11 11:30:00.170106 master-1 kubenswrapper[4771]: I1011 11:30:00.170045 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts"] Oct 11 11:30:00.171039 master-1 kubenswrapper[4771]: E1011 11:30:00.170487 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerName="extract-content" Oct 11 11:30:00.171039 master-1 kubenswrapper[4771]: I1011 11:30:00.170502 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerName="extract-content" Oct 11 11:30:00.171039 master-1 kubenswrapper[4771]: E1011 11:30:00.170522 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerName="extract-utilities" Oct 11 11:30:00.171039 master-1 kubenswrapper[4771]: I1011 11:30:00.170528 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerName="extract-utilities" Oct 11 11:30:00.171039 master-1 kubenswrapper[4771]: E1011 11:30:00.170544 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerName="registry-server" Oct 11 11:30:00.171039 master-1 kubenswrapper[4771]: I1011 11:30:00.170551 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerName="registry-server" Oct 11 11:30:00.171039 master-1 kubenswrapper[4771]: I1011 11:30:00.170744 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="99934513-1a05-40c6-8b59-cca04d84b0cd" containerName="registry-server" Oct 11 11:30:00.171723 master-1 kubenswrapper[4771]: I1011 11:30:00.171538 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.174386 master-1 kubenswrapper[4771]: I1011 11:30:00.174339 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-operator-lifecycle-manager"/"collect-profiles-config" Oct 11 11:30:00.175084 master-1 kubenswrapper[4771]: I1011 11:30:00.174566 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-operator-lifecycle-manager"/"pprof-cert" Oct 11 11:30:00.175280 master-1 kubenswrapper[4771]: I1011 11:30:00.175255 4771 reflector.go:368] Caches populated for *v1.Secret from object-"openshift-operator-lifecycle-manager"/"collect-profiles-dockercfg-hbjq2" Oct 11 11:30:00.187244 master-1 kubenswrapper[4771]: I1011 11:30:00.186868 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts"] Oct 11 11:30:00.334928 master-1 kubenswrapper[4771]: I1011 11:30:00.334844 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/a19b5781-05b5-4109-a6e0-c54746c813a6-secret-volume\") pod \"collect-profiles-29336370-9vpts\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.335205 master-1 kubenswrapper[4771]: I1011 11:30:00.334982 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r22x\" (UniqueName: \"kubernetes.io/projected/a19b5781-05b5-4109-a6e0-c54746c813a6-kube-api-access-7r22x\") pod \"collect-profiles-29336370-9vpts\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.335205 master-1 kubenswrapper[4771]: I1011 11:30:00.335041 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a19b5781-05b5-4109-a6e0-c54746c813a6-config-volume\") pod \"collect-profiles-29336370-9vpts\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.436824 master-1 kubenswrapper[4771]: I1011 11:30:00.436647 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a19b5781-05b5-4109-a6e0-c54746c813a6-config-volume\") pod \"collect-profiles-29336370-9vpts\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.437072 master-1 kubenswrapper[4771]: I1011 11:30:00.436820 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/a19b5781-05b5-4109-a6e0-c54746c813a6-secret-volume\") pod \"collect-profiles-29336370-9vpts\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.437072 master-1 kubenswrapper[4771]: I1011 11:30:00.436980 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-7r22x\" (UniqueName: \"kubernetes.io/projected/a19b5781-05b5-4109-a6e0-c54746c813a6-kube-api-access-7r22x\") pod \"collect-profiles-29336370-9vpts\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.438649 master-1 kubenswrapper[4771]: I1011 11:30:00.438580 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a19b5781-05b5-4109-a6e0-c54746c813a6-config-volume\") pod \"collect-profiles-29336370-9vpts\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.443020 master-1 kubenswrapper[4771]: I1011 11:30:00.442887 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/a19b5781-05b5-4109-a6e0-c54746c813a6-secret-volume\") pod \"collect-profiles-29336370-9vpts\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.469696 master-1 kubenswrapper[4771]: I1011 11:30:00.469623 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r22x\" (UniqueName: \"kubernetes.io/projected/a19b5781-05b5-4109-a6e0-c54746c813a6-kube-api-access-7r22x\") pod \"collect-profiles-29336370-9vpts\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:00.515811 master-1 kubenswrapper[4771]: I1011 11:30:00.515737 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:01.041214 master-1 kubenswrapper[4771]: I1011 11:30:01.041149 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts"] Oct 11 11:30:01.949803 master-1 kubenswrapper[4771]: I1011 11:30:01.949744 4771 generic.go:334] "Generic (PLEG): container finished" podID="a19b5781-05b5-4109-a6e0-c54746c813a6" containerID="c64fb4f8711dfea2086b2ddbeb235c0580214bb088c49212c033f84f4ec3bced" exitCode=0 Oct 11 11:30:01.949803 master-1 kubenswrapper[4771]: I1011 11:30:01.949806 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" event={"ID":"a19b5781-05b5-4109-a6e0-c54746c813a6","Type":"ContainerDied","Data":"c64fb4f8711dfea2086b2ddbeb235c0580214bb088c49212c033f84f4ec3bced"} Oct 11 11:30:01.950780 master-1 kubenswrapper[4771]: I1011 11:30:01.949846 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" event={"ID":"a19b5781-05b5-4109-a6e0-c54746c813a6","Type":"ContainerStarted","Data":"6b905a90da79c6fe1c0f82c63b036fdb56d7a5d66007ecbfadd8bd681f8492a9"} Oct 11 11:30:03.508979 master-1 kubenswrapper[4771]: I1011 11:30:03.508917 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:30:03.634707 master-1 kubenswrapper[4771]: I1011 11:30:03.634615 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a19b5781-05b5-4109-a6e0-c54746c813a6-config-volume\") pod \"a19b5781-05b5-4109-a6e0-c54746c813a6\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " Oct 11 11:30:03.635060 master-1 kubenswrapper[4771]: I1011 11:30:03.634727 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r22x\" (UniqueName: \"kubernetes.io/projected/a19b5781-05b5-4109-a6e0-c54746c813a6-kube-api-access-7r22x\") pod \"a19b5781-05b5-4109-a6e0-c54746c813a6\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " Oct 11 11:30:03.635060 master-1 kubenswrapper[4771]: I1011 11:30:03.634802 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/a19b5781-05b5-4109-a6e0-c54746c813a6-secret-volume\") pod \"a19b5781-05b5-4109-a6e0-c54746c813a6\" (UID: \"a19b5781-05b5-4109-a6e0-c54746c813a6\") " Oct 11 11:30:03.635403 master-1 kubenswrapper[4771]: I1011 11:30:03.635317 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19b5781-05b5-4109-a6e0-c54746c813a6-config-volume" (OuterVolumeSpecName: "config-volume") pod "a19b5781-05b5-4109-a6e0-c54746c813a6" (UID: "a19b5781-05b5-4109-a6e0-c54746c813a6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/configmap", VolumeGidValue "" Oct 11 11:30:03.640567 master-1 kubenswrapper[4771]: I1011 11:30:03.638670 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19b5781-05b5-4109-a6e0-c54746c813a6-secret-volume" (OuterVolumeSpecName: "secret-volume") pod "a19b5781-05b5-4109-a6e0-c54746c813a6" (UID: "a19b5781-05b5-4109-a6e0-c54746c813a6"). InnerVolumeSpecName "secret-volume". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:30:03.640567 master-1 kubenswrapper[4771]: I1011 11:30:03.639113 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19b5781-05b5-4109-a6e0-c54746c813a6-kube-api-access-7r22x" (OuterVolumeSpecName: "kube-api-access-7r22x") pod "a19b5781-05b5-4109-a6e0-c54746c813a6" (UID: "a19b5781-05b5-4109-a6e0-c54746c813a6"). InnerVolumeSpecName "kube-api-access-7r22x". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:30:03.737789 master-1 kubenswrapper[4771]: I1011 11:30:03.737717 4771 reconciler_common.go:293] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a19b5781-05b5-4109-a6e0-c54746c813a6-config-volume\") on node \"master-1\" DevicePath \"\"" Oct 11 11:30:03.737789 master-1 kubenswrapper[4771]: I1011 11:30:03.737771 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-7r22x\" (UniqueName: \"kubernetes.io/projected/a19b5781-05b5-4109-a6e0-c54746c813a6-kube-api-access-7r22x\") on node \"master-1\" DevicePath \"\"" Oct 11 11:30:03.737789 master-1 kubenswrapper[4771]: I1011 11:30:03.737786 4771 reconciler_common.go:293] "Volume detached for volume \"secret-volume\" (UniqueName: \"kubernetes.io/secret/a19b5781-05b5-4109-a6e0-c54746c813a6-secret-volume\") on node \"master-1\" DevicePath \"\"" Oct 11 11:30:03.971295 master-1 kubenswrapper[4771]: I1011 11:30:03.971082 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" event={"ID":"a19b5781-05b5-4109-a6e0-c54746c813a6","Type":"ContainerDied","Data":"6b905a90da79c6fe1c0f82c63b036fdb56d7a5d66007ecbfadd8bd681f8492a9"} Oct 11 11:30:03.971295 master-1 kubenswrapper[4771]: I1011 11:30:03.971170 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b905a90da79c6fe1c0f82c63b036fdb56d7a5d66007ecbfadd8bd681f8492a9" Oct 11 11:30:03.971295 master-1 kubenswrapper[4771]: I1011 11:30:03.971121 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-operator-lifecycle-manager/collect-profiles-29336370-9vpts" Oct 11 11:31:13.770546 master-1 kubenswrapper[4771]: I1011 11:31:13.770442 4771 generic.go:334] "Generic (PLEG): container finished" podID="fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" containerID="71b6cbea99b537fb97df8cac56078c84bd8fb6f72db7edf36ac99e71415f9777" exitCode=0 Oct 11 11:31:13.770546 master-1 kubenswrapper[4771]: I1011 11:31:13.770528 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/telemetry-dataplane-edpm-rdjt7" event={"ID":"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55","Type":"ContainerDied","Data":"71b6cbea99b537fb97df8cac56078c84bd8fb6f72db7edf36ac99e71415f9777"} Oct 11 11:31:15.392560 master-1 kubenswrapper[4771]: I1011 11:31:15.392466 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:31:15.541147 master-1 kubenswrapper[4771]: I1011 11:31:15.541042 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"telemetry-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-telemetry-combined-ca-bundle\") pod \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " Oct 11 11:31:15.541147 master-1 kubenswrapper[4771]: I1011 11:31:15.541130 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ceilometer-compute-config-data-0\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-0\") pod \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " Oct 11 11:31:15.541602 master-1 kubenswrapper[4771]: I1011 11:31:15.541208 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzw24\" (UniqueName: \"kubernetes.io/projected/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-kube-api-access-hzw24\") pod \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " Oct 11 11:31:15.541602 master-1 kubenswrapper[4771]: I1011 11:31:15.541309 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ssh-key\") pod \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " Oct 11 11:31:15.541602 master-1 kubenswrapper[4771]: I1011 11:31:15.541362 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ceilometer-compute-config-data-2\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-2\") pod \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " Oct 11 11:31:15.541602 master-1 kubenswrapper[4771]: I1011 11:31:15.541440 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"ceilometer-compute-config-data-1\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-1\") pod \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " Oct 11 11:31:15.541602 master-1 kubenswrapper[4771]: I1011 11:31:15.541467 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-inventory\") pod \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\" (UID: \"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55\") " Oct 11 11:31:15.546779 master-1 kubenswrapper[4771]: I1011 11:31:15.546512 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-kube-api-access-hzw24" (OuterVolumeSpecName: "kube-api-access-hzw24") pod "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" (UID: "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55"). InnerVolumeSpecName "kube-api-access-hzw24". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:31:15.547413 master-1 kubenswrapper[4771]: I1011 11:31:15.547292 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-telemetry-combined-ca-bundle" (OuterVolumeSpecName: "telemetry-combined-ca-bundle") pod "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" (UID: "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55"). InnerVolumeSpecName "telemetry-combined-ca-bundle". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:31:15.565547 master-1 kubenswrapper[4771]: I1011 11:31:15.565433 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-1" (OuterVolumeSpecName: "ceilometer-compute-config-data-1") pod "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" (UID: "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55"). InnerVolumeSpecName "ceilometer-compute-config-data-1". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:31:15.570055 master-1 kubenswrapper[4771]: I1011 11:31:15.569998 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-inventory" (OuterVolumeSpecName: "inventory") pod "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" (UID: "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55"). InnerVolumeSpecName "inventory". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:31:15.571222 master-1 kubenswrapper[4771]: I1011 11:31:15.571148 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-2" (OuterVolumeSpecName: "ceilometer-compute-config-data-2") pod "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" (UID: "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55"). InnerVolumeSpecName "ceilometer-compute-config-data-2". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:31:15.571747 master-1 kubenswrapper[4771]: I1011 11:31:15.571708 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-0" (OuterVolumeSpecName: "ceilometer-compute-config-data-0") pod "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" (UID: "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55"). InnerVolumeSpecName "ceilometer-compute-config-data-0". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:31:15.598144 master-1 kubenswrapper[4771]: I1011 11:31:15.598073 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ssh-key" (OuterVolumeSpecName: "ssh-key") pod "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" (UID: "fb1d9f67-e3e9-4545-9e5d-07f2c213fe55"). InnerVolumeSpecName "ssh-key". PluginName "kubernetes.io/secret", VolumeGidValue "" Oct 11 11:31:15.646387 master-1 kubenswrapper[4771]: I1011 11:31:15.646268 4771 reconciler_common.go:293] "Volume detached for volume \"ssh-key\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ssh-key\") on node \"master-1\" DevicePath \"\"" Oct 11 11:31:15.646387 master-1 kubenswrapper[4771]: I1011 11:31:15.646336 4771 reconciler_common.go:293] "Volume detached for volume \"ceilometer-compute-config-data-2\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-2\") on node \"master-1\" DevicePath \"\"" Oct 11 11:31:15.646975 master-1 kubenswrapper[4771]: I1011 11:31:15.646594 4771 reconciler_common.go:293] "Volume detached for volume \"ceilometer-compute-config-data-1\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-1\") on node \"master-1\" DevicePath \"\"" Oct 11 11:31:15.646975 master-1 kubenswrapper[4771]: I1011 11:31:15.646626 4771 reconciler_common.go:293] "Volume detached for volume \"inventory\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-inventory\") on node \"master-1\" DevicePath \"\"" Oct 11 11:31:15.646975 master-1 kubenswrapper[4771]: I1011 11:31:15.646653 4771 reconciler_common.go:293] "Volume detached for volume \"telemetry-combined-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-telemetry-combined-ca-bundle\") on node \"master-1\" DevicePath \"\"" Oct 11 11:31:15.646975 master-1 kubenswrapper[4771]: I1011 11:31:15.646676 4771 reconciler_common.go:293] "Volume detached for volume \"ceilometer-compute-config-data-0\" (UniqueName: \"kubernetes.io/secret/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-ceilometer-compute-config-data-0\") on node \"master-1\" DevicePath \"\"" Oct 11 11:31:15.646975 master-1 kubenswrapper[4771]: I1011 11:31:15.646698 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-hzw24\" (UniqueName: \"kubernetes.io/projected/fb1d9f67-e3e9-4545-9e5d-07f2c213fe55-kube-api-access-hzw24\") on node \"master-1\" DevicePath \"\"" Oct 11 11:31:15.789535 master-1 kubenswrapper[4771]: I1011 11:31:15.789451 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openstack/telemetry-dataplane-edpm-rdjt7" event={"ID":"fb1d9f67-e3e9-4545-9e5d-07f2c213fe55","Type":"ContainerDied","Data":"d621ccd58559916f2503ebdd3bbdd04a63949a337855cab279f0c0f2f83d42a6"} Oct 11 11:31:15.789535 master-1 kubenswrapper[4771]: I1011 11:31:15.789508 4771 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d621ccd58559916f2503ebdd3bbdd04a63949a337855cab279f0c0f2f83d42a6" Oct 11 11:31:15.789835 master-1 kubenswrapper[4771]: I1011 11:31:15.789563 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openstack/telemetry-dataplane-edpm-rdjt7" Oct 11 11:31:43.877474 master-1 kubenswrapper[4771]: I1011 11:31:43.877267 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fg6sc/must-gather-t7wkh"] Oct 11 11:31:43.878608 master-1 kubenswrapper[4771]: E1011 11:31:43.877945 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="a19b5781-05b5-4109-a6e0-c54746c813a6" containerName="collect-profiles" Oct 11 11:31:43.878608 master-1 kubenswrapper[4771]: I1011 11:31:43.877969 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19b5781-05b5-4109-a6e0-c54746c813a6" containerName="collect-profiles" Oct 11 11:31:43.878608 master-1 kubenswrapper[4771]: E1011 11:31:43.877992 4771 cpu_manager.go:410] "RemoveStaleState: removing container" podUID="fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" containerName="telemetry-dataplane-edpm" Oct 11 11:31:43.878608 master-1 kubenswrapper[4771]: I1011 11:31:43.878006 4771 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" containerName="telemetry-dataplane-edpm" Oct 11 11:31:43.878608 master-1 kubenswrapper[4771]: I1011 11:31:43.878285 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="a19b5781-05b5-4109-a6e0-c54746c813a6" containerName="collect-profiles" Oct 11 11:31:43.878608 master-1 kubenswrapper[4771]: I1011 11:31:43.878313 4771 memory_manager.go:354] "RemoveStaleState removing state" podUID="fb1d9f67-e3e9-4545-9e5d-07f2c213fe55" containerName="telemetry-dataplane-edpm" Oct 11 11:31:43.880280 master-1 kubenswrapper[4771]: I1011 11:31:43.880112 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg6sc/must-gather-t7wkh" Oct 11 11:31:43.883890 master-1 kubenswrapper[4771]: I1011 11:31:43.883838 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-must-gather-fg6sc"/"kube-root-ca.crt" Oct 11 11:31:43.884167 master-1 kubenswrapper[4771]: I1011 11:31:43.884127 4771 reflector.go:368] Caches populated for *v1.ConfigMap from object-"openshift-must-gather-fg6sc"/"openshift-service-ca.crt" Oct 11 11:31:43.893556 master-1 kubenswrapper[4771]: I1011 11:31:43.893003 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg6sc/must-gather-t7wkh"] Oct 11 11:31:44.030389 master-1 kubenswrapper[4771]: I1011 11:31:44.027047 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcstv\" (UniqueName: \"kubernetes.io/projected/47607144-c636-4055-b3d3-fc54a36cf0e5-kube-api-access-tcstv\") pod \"must-gather-t7wkh\" (UID: \"47607144-c636-4055-b3d3-fc54a36cf0e5\") " pod="openshift-must-gather-fg6sc/must-gather-t7wkh" Oct 11 11:31:44.030389 master-1 kubenswrapper[4771]: I1011 11:31:44.027147 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/47607144-c636-4055-b3d3-fc54a36cf0e5-must-gather-output\") pod \"must-gather-t7wkh\" (UID: \"47607144-c636-4055-b3d3-fc54a36cf0e5\") " pod="openshift-must-gather-fg6sc/must-gather-t7wkh" Oct 11 11:31:44.132455 master-1 kubenswrapper[4771]: I1011 11:31:44.128674 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-tcstv\" (UniqueName: \"kubernetes.io/projected/47607144-c636-4055-b3d3-fc54a36cf0e5-kube-api-access-tcstv\") pod \"must-gather-t7wkh\" (UID: \"47607144-c636-4055-b3d3-fc54a36cf0e5\") " pod="openshift-must-gather-fg6sc/must-gather-t7wkh" Oct 11 11:31:44.132455 master-1 kubenswrapper[4771]: I1011 11:31:44.128745 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/47607144-c636-4055-b3d3-fc54a36cf0e5-must-gather-output\") pod \"must-gather-t7wkh\" (UID: \"47607144-c636-4055-b3d3-fc54a36cf0e5\") " pod="openshift-must-gather-fg6sc/must-gather-t7wkh" Oct 11 11:31:44.132455 master-1 kubenswrapper[4771]: I1011 11:31:44.129276 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/47607144-c636-4055-b3d3-fc54a36cf0e5-must-gather-output\") pod \"must-gather-t7wkh\" (UID: \"47607144-c636-4055-b3d3-fc54a36cf0e5\") " pod="openshift-must-gather-fg6sc/must-gather-t7wkh" Oct 11 11:31:44.164380 master-1 kubenswrapper[4771]: I1011 11:31:44.164031 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcstv\" (UniqueName: \"kubernetes.io/projected/47607144-c636-4055-b3d3-fc54a36cf0e5-kube-api-access-tcstv\") pod \"must-gather-t7wkh\" (UID: \"47607144-c636-4055-b3d3-fc54a36cf0e5\") " pod="openshift-must-gather-fg6sc/must-gather-t7wkh" Oct 11 11:31:44.202380 master-1 kubenswrapper[4771]: I1011 11:31:44.201836 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg6sc/must-gather-t7wkh" Oct 11 11:31:44.681538 master-1 kubenswrapper[4771]: I1011 11:31:44.681064 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg6sc/must-gather-t7wkh"] Oct 11 11:31:45.113288 master-1 kubenswrapper[4771]: I1011 11:31:45.113185 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg6sc/must-gather-t7wkh" event={"ID":"47607144-c636-4055-b3d3-fc54a36cf0e5","Type":"ContainerStarted","Data":"c7c3fdea0c455c10846c5aae6c73c93ccd9ceafbba080e5eefb34abdb563b139"} Oct 11 11:31:50.177189 master-1 kubenswrapper[4771]: I1011 11:31:50.177106 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg6sc/must-gather-t7wkh" event={"ID":"47607144-c636-4055-b3d3-fc54a36cf0e5","Type":"ContainerStarted","Data":"4cf5cae6f0470c7fc3e748af0811d9656faaa6cb7a892484e0fe05cbc22df70d"} Oct 11 11:31:50.177189 master-1 kubenswrapper[4771]: I1011 11:31:50.177183 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg6sc/must-gather-t7wkh" event={"ID":"47607144-c636-4055-b3d3-fc54a36cf0e5","Type":"ContainerStarted","Data":"0d94abe58a407a6d13fb72659080dd3e6c3d55607b6c894edcf614c3d5e380e6"} Oct 11 11:31:50.207635 master-1 kubenswrapper[4771]: I1011 11:31:50.207514 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-fg6sc/must-gather-t7wkh" podStartSLOduration=3.00724719 podStartE2EDuration="7.207490681s" podCreationTimestamp="2025-10-11 11:31:43 +0000 UTC" firstStartedPulling="2025-10-11 11:31:44.697013721 +0000 UTC m=+3936.671240162" lastFinishedPulling="2025-10-11 11:31:48.897257202 +0000 UTC m=+3940.871483653" observedRunningTime="2025-10-11 11:31:50.199497762 +0000 UTC m=+3942.173724213" watchObservedRunningTime="2025-10-11 11:31:50.207490681 +0000 UTC m=+3942.181717112" Oct 11 11:31:51.520488 master-1 kubenswrapper[4771]: I1011 11:31:51.520423 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-nmstate_nmstate-handler-djsq6_3e92427e-68a9-4496-9578-a0386bd5f5b3/nmstate-handler/0.log" Oct 11 11:31:52.202899 master-1 kubenswrapper[4771]: I1011 11:31:52.202838 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-guard-master-1_3fc4970d-4f34-4fc6-9791-6218f8e42eb9/guard/0.log" Oct 11 11:31:52.658186 master-1 kubenswrapper[4771]: I1011 11:31:52.655761 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcdctl/0.log" Oct 11 11:31:52.762876 master-1 kubenswrapper[4771]: I1011 11:31:52.762822 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd/0.log" Oct 11 11:31:52.786864 master-1 kubenswrapper[4771]: I1011 11:31:52.786818 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-metrics/0.log" Oct 11 11:31:52.824123 master-1 kubenswrapper[4771]: I1011 11:31:52.820756 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-readyz/0.log" Oct 11 11:31:52.853396 master-1 kubenswrapper[4771]: I1011 11:31:52.853342 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-rev/0.log" Oct 11 11:31:52.872638 master-1 kubenswrapper[4771]: I1011 11:31:52.872600 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/setup/0.log" Oct 11 11:31:52.922078 master-1 kubenswrapper[4771]: I1011 11:31:52.920636 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-ensure-env-vars/0.log" Oct 11 11:31:52.958210 master-1 kubenswrapper[4771]: I1011 11:31:52.958143 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-resources-copy/0.log" Oct 11 11:31:53.518445 master-1 kubenswrapper[4771]: I1011 11:31:53.518348 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_installer-10-master-1_8d7775e5-5c08-4eef-84bf-8995a11eb190/installer/0.log" Oct 11 11:31:53.715449 master-1 kubenswrapper[4771]: I1011 11:31:53.715283 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_revision-pruner-10-master-1_a47a3143-b015-49c8-a15d-678e348b64e8/pruner/0.log" Oct 11 11:31:53.914248 master-1 kubenswrapper[4771]: I1011 11:31:53.914187 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/controller/0.log" Oct 11 11:31:54.213575 master-1 kubenswrapper[4771]: I1011 11:31:54.209613 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-marketplace/redhat-marketplace-pzfn2"] Oct 11 11:31:54.215628 master-1 kubenswrapper[4771]: I1011 11:31:54.215570 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.220453 master-1 kubenswrapper[4771]: I1011 11:31:54.220376 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-authentication_oauth-openshift-6fccd5ccc-txx8d_1ec66eef-540b-4e9a-b63a-02d662224040/oauth-openshift/0.log" Oct 11 11:31:54.228064 master-1 kubenswrapper[4771]: I1011 11:31:54.227641 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-pzfn2"] Oct 11 11:31:54.368830 master-1 kubenswrapper[4771]: I1011 11:31:54.368758 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-catalog-content\") pod \"redhat-marketplace-pzfn2\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.369077 master-1 kubenswrapper[4771]: I1011 11:31:54.368873 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdg2k\" (UniqueName: \"kubernetes.io/projected/1f01744c-374a-4f9f-94d3-d745ecff926e-kube-api-access-gdg2k\") pod \"redhat-marketplace-pzfn2\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.369248 master-1 kubenswrapper[4771]: I1011 11:31:54.369194 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-utilities\") pod \"redhat-marketplace-pzfn2\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.470636 master-1 kubenswrapper[4771]: I1011 11:31:54.470580 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-gdg2k\" (UniqueName: \"kubernetes.io/projected/1f01744c-374a-4f9f-94d3-d745ecff926e-kube-api-access-gdg2k\") pod \"redhat-marketplace-pzfn2\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.470873 master-1 kubenswrapper[4771]: I1011 11:31:54.470677 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-utilities\") pod \"redhat-marketplace-pzfn2\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.470873 master-1 kubenswrapper[4771]: I1011 11:31:54.470716 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-catalog-content\") pod \"redhat-marketplace-pzfn2\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.471218 master-1 kubenswrapper[4771]: I1011 11:31:54.471193 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-catalog-content\") pod \"redhat-marketplace-pzfn2\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.472161 master-1 kubenswrapper[4771]: I1011 11:31:54.471441 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-utilities\") pod \"redhat-marketplace-pzfn2\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.506529 master-1 kubenswrapper[4771]: I1011 11:31:54.506487 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdg2k\" (UniqueName: \"kubernetes.io/projected/1f01744c-374a-4f9f-94d3-d745ecff926e-kube-api-access-gdg2k\") pod \"redhat-marketplace-pzfn2\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.534950 master-1 kubenswrapper[4771]: I1011 11:31:54.534892 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:31:54.945242 master-1 kubenswrapper[4771]: I1011 11:31:54.945089 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fg6sc/master-1-debug-4v5zr"] Oct 11 11:31:54.949060 master-1 kubenswrapper[4771]: I1011 11:31:54.949020 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" Oct 11 11:31:55.080608 master-1 kubenswrapper[4771]: W1011 11:31:55.080553 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f01744c_374a_4f9f_94d3_d745ecff926e.slice/crio-2f60b45e5cf1e747316ed60ad595c1d86741ef8181ad7f4c0d4a20981df8b87b WatchSource:0}: Error finding container 2f60b45e5cf1e747316ed60ad595c1d86741ef8181ad7f4c0d4a20981df8b87b: Status 404 returned error can't find the container with id 2f60b45e5cf1e747316ed60ad595c1d86741ef8181ad7f4c0d4a20981df8b87b Oct 11 11:31:55.081311 master-1 kubenswrapper[4771]: I1011 11:31:55.081263 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7149351f-3d07-43c4-9f74-1b9052a9c04b-host\") pod \"master-1-debug-4v5zr\" (UID: \"7149351f-3d07-43c4-9f74-1b9052a9c04b\") " pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" Oct 11 11:31:55.082031 master-1 kubenswrapper[4771]: I1011 11:31:55.082001 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24h9\" (UniqueName: \"kubernetes.io/projected/7149351f-3d07-43c4-9f74-1b9052a9c04b-kube-api-access-s24h9\") pod \"master-1-debug-4v5zr\" (UID: \"7149351f-3d07-43c4-9f74-1b9052a9c04b\") " pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" Oct 11 11:31:55.084229 master-1 kubenswrapper[4771]: I1011 11:31:55.084038 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/redhat-marketplace-pzfn2"] Oct 11 11:31:55.148198 master-1 kubenswrapper[4771]: I1011 11:31:55.148129 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/frr/0.log" Oct 11 11:31:55.163641 master-1 kubenswrapper[4771]: I1011 11:31:55.163576 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/reloader/0.log" Oct 11 11:31:55.180861 master-1 kubenswrapper[4771]: I1011 11:31:55.180824 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/frr-metrics/0.log" Oct 11 11:31:55.185395 master-1 kubenswrapper[4771]: I1011 11:31:55.185333 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-s24h9\" (UniqueName: \"kubernetes.io/projected/7149351f-3d07-43c4-9f74-1b9052a9c04b-kube-api-access-s24h9\") pod \"master-1-debug-4v5zr\" (UID: \"7149351f-3d07-43c4-9f74-1b9052a9c04b\") " pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" Oct 11 11:31:55.185592 master-1 kubenswrapper[4771]: I1011 11:31:55.185566 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7149351f-3d07-43c4-9f74-1b9052a9c04b-host\") pod \"master-1-debug-4v5zr\" (UID: \"7149351f-3d07-43c4-9f74-1b9052a9c04b\") " pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" Oct 11 11:31:55.185717 master-1 kubenswrapper[4771]: I1011 11:31:55.185696 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7149351f-3d07-43c4-9f74-1b9052a9c04b-host\") pod \"master-1-debug-4v5zr\" (UID: \"7149351f-3d07-43c4-9f74-1b9052a9c04b\") " pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" Oct 11 11:31:55.197169 master-1 kubenswrapper[4771]: I1011 11:31:55.197061 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/kube-rbac-proxy/0.log" Oct 11 11:31:55.214210 master-1 kubenswrapper[4771]: I1011 11:31:55.214156 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24h9\" (UniqueName: \"kubernetes.io/projected/7149351f-3d07-43c4-9f74-1b9052a9c04b-kube-api-access-s24h9\") pod \"master-1-debug-4v5zr\" (UID: \"7149351f-3d07-43c4-9f74-1b9052a9c04b\") " pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" Oct 11 11:31:55.217850 master-1 kubenswrapper[4771]: I1011 11:31:55.217817 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/kube-rbac-proxy-frr/0.log" Oct 11 11:31:55.229179 master-1 kubenswrapper[4771]: I1011 11:31:55.229126 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-pzfn2" event={"ID":"1f01744c-374a-4f9f-94d3-d745ecff926e","Type":"ContainerStarted","Data":"2f60b45e5cf1e747316ed60ad595c1d86741ef8181ad7f4c0d4a20981df8b87b"} Oct 11 11:31:55.231513 master-1 kubenswrapper[4771]: I1011 11:31:55.231491 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/cp-frr-files/0.log" Oct 11 11:31:55.247426 master-1 kubenswrapper[4771]: I1011 11:31:55.247389 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/cp-reloader/0.log" Oct 11 11:31:55.272441 master-1 kubenswrapper[4771]: I1011 11:31:55.272401 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/cp-metrics/0.log" Oct 11 11:31:55.279685 master-1 kubenswrapper[4771]: I1011 11:31:55.279635 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" Oct 11 11:31:55.374031 master-1 kubenswrapper[4771]: W1011 11:31:55.373961 4771 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7149351f_3d07_43c4_9f74_1b9052a9c04b.slice/crio-244d5ef958bc640d5db8cdc87aab15dd1ee54264c7627d45ca033605a2bc2af8 WatchSource:0}: Error finding container 244d5ef958bc640d5db8cdc87aab15dd1ee54264c7627d45ca033605a2bc2af8: Status 404 returned error can't find the container with id 244d5ef958bc640d5db8cdc87aab15dd1ee54264c7627d45ca033605a2bc2af8 Oct 11 11:31:56.065845 master-1 kubenswrapper[4771]: I1011 11:31:56.065793 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_speaker-524kt_72f01dc6-72cd-4eb0-8039-57150e0758bf/speaker/0.log" Oct 11 11:31:56.076611 master-1 kubenswrapper[4771]: I1011 11:31:56.076571 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_speaker-524kt_72f01dc6-72cd-4eb0-8039-57150e0758bf/kube-rbac-proxy/0.log" Oct 11 11:31:56.239401 master-1 kubenswrapper[4771]: I1011 11:31:56.239093 4771 generic.go:334] "Generic (PLEG): container finished" podID="1f01744c-374a-4f9f-94d3-d745ecff926e" containerID="dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4" exitCode=0 Oct 11 11:31:56.239401 master-1 kubenswrapper[4771]: I1011 11:31:56.239162 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-pzfn2" event={"ID":"1f01744c-374a-4f9f-94d3-d745ecff926e","Type":"ContainerDied","Data":"dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4"} Oct 11 11:31:56.245770 master-1 kubenswrapper[4771]: I1011 11:31:56.245323 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" event={"ID":"7149351f-3d07-43c4-9f74-1b9052a9c04b","Type":"ContainerStarted","Data":"244d5ef958bc640d5db8cdc87aab15dd1ee54264c7627d45ca033605a2bc2af8"} Oct 11 11:31:56.334516 master-1 kubenswrapper[4771]: I1011 11:31:56.334381 4771 kubelet.go:2421] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj"] Oct 11 11:31:56.335717 master-1 kubenswrapper[4771]: I1011 11:31:56.335698 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.361330 master-1 kubenswrapper[4771]: I1011 11:31:56.361257 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj"] Oct 11 11:31:56.410042 master-1 kubenswrapper[4771]: I1011 11:31:56.409969 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-proc\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.411997 master-1 kubenswrapper[4771]: I1011 11:31:56.410060 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-podres\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.411997 master-1 kubenswrapper[4771]: I1011 11:31:56.410129 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-lib-modules\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.411997 master-1 kubenswrapper[4771]: I1011 11:31:56.410153 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-sys\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.411997 master-1 kubenswrapper[4771]: I1011 11:31:56.410281 4771 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9knr\" (UniqueName: \"kubernetes.io/projected/f5e43e99-33a2-4a88-b233-24e88075765a-kube-api-access-n9knr\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.513247 master-1 kubenswrapper[4771]: I1011 11:31:56.512091 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-lib-modules\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.513247 master-1 kubenswrapper[4771]: I1011 11:31:56.512160 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-sys\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.513247 master-1 kubenswrapper[4771]: I1011 11:31:56.512194 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"kube-api-access-n9knr\" (UniqueName: \"kubernetes.io/projected/f5e43e99-33a2-4a88-b233-24e88075765a-kube-api-access-n9knr\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.513247 master-1 kubenswrapper[4771]: I1011 11:31:56.512279 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-proc\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.513247 master-1 kubenswrapper[4771]: I1011 11:31:56.512419 4771 reconciler_common.go:218] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-podres\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.513247 master-1 kubenswrapper[4771]: I1011 11:31:56.512822 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-sys\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.515204 master-1 kubenswrapper[4771]: I1011 11:31:56.514017 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-proc\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.515204 master-1 kubenswrapper[4771]: I1011 11:31:56.514234 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-podres\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.515760 master-1 kubenswrapper[4771]: I1011 11:31:56.515565 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5e43e99-33a2-4a88-b233-24e88075765a-lib-modules\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.531287 master-1 kubenswrapper[4771]: I1011 11:31:56.531227 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5ddb89f76-z5t6x_04cd4a19-2532-43d1-9144-1f59d9e52d19/router/1.log" Oct 11 11:31:56.532075 master-1 kubenswrapper[4771]: I1011 11:31:56.532045 4771 operation_generator.go:637] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9knr\" (UniqueName: \"kubernetes.io/projected/f5e43e99-33a2-4a88-b233-24e88075765a-kube-api-access-n9knr\") pod \"perf-node-gather-daemonset-czfkj\" (UID: \"f5e43e99-33a2-4a88-b233-24e88075765a\") " pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:56.533323 master-1 kubenswrapper[4771]: I1011 11:31:56.533285 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5ddb89f76-z5t6x_04cd4a19-2532-43d1-9144-1f59d9e52d19/router/2.log" Oct 11 11:31:56.658468 master-1 kubenswrapper[4771]: I1011 11:31:56.658313 4771 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:57.123537 master-1 kubenswrapper[4771]: I1011 11:31:57.123469 4771 kubelet.go:2428] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj"] Oct 11 11:31:57.255663 master-1 kubenswrapper[4771]: I1011 11:31:57.255590 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" event={"ID":"f5e43e99-33a2-4a88-b233-24e88075765a","Type":"ContainerStarted","Data":"ce8994224e0197fb0967d2b336c8c2b35ef3ca6d6a05f9374842d61ca480b6fd"} Oct 11 11:31:57.405412 master-1 kubenswrapper[4771]: I1011 11:31:57.405272 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-oauth-apiserver_apiserver-68f4c55ff4-mmqll_1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d/oauth-apiserver/0.log" Oct 11 11:31:57.429103 master-1 kubenswrapper[4771]: I1011 11:31:57.429070 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-oauth-apiserver_apiserver-68f4c55ff4-mmqll_1a8be6e8-cddf-46d6-a1e4-f0bdc0ae7f6d/fix-audit-permissions/0.log" Oct 11 11:31:58.269132 master-1 kubenswrapper[4771]: I1011 11:31:58.269069 4771 generic.go:334] "Generic (PLEG): container finished" podID="1f01744c-374a-4f9f-94d3-d745ecff926e" containerID="75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813" exitCode=0 Oct 11 11:31:58.269695 master-1 kubenswrapper[4771]: I1011 11:31:58.269141 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-pzfn2" event={"ID":"1f01744c-374a-4f9f-94d3-d745ecff926e","Type":"ContainerDied","Data":"75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813"} Oct 11 11:31:58.273545 master-1 kubenswrapper[4771]: I1011 11:31:58.273494 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" event={"ID":"f5e43e99-33a2-4a88-b233-24e88075765a","Type":"ContainerStarted","Data":"53f529ee1056451ac6b194f4ca6d45dc78260e62b74894b08955ecec8d643847"} Oct 11 11:31:58.273767 master-1 kubenswrapper[4771]: I1011 11:31:58.273742 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:31:58.342823 master-1 kubenswrapper[4771]: I1011 11:31:58.342521 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" podStartSLOduration=2.342484637 podStartE2EDuration="2.342484637s" podCreationTimestamp="2025-10-11 11:31:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-10-11 11:31:58.329984229 +0000 UTC m=+3950.304210710" watchObservedRunningTime="2025-10-11 11:31:58.342484637 +0000 UTC m=+3950.316711118" Oct 11 11:32:00.711338 master-1 kubenswrapper[4771]: I1011 11:32:00.711221 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/cluster-cloud-controller-manager/0.log" Oct 11 11:32:00.738143 master-1 kubenswrapper[4771]: I1011 11:32:00.738085 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/config-sync-controllers/0.log" Oct 11 11:32:00.766988 master-1 kubenswrapper[4771]: I1011 11:32:00.766889 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/3.log" Oct 11 11:32:00.767537 master-1 kubenswrapper[4771]: I1011 11:32:00.767317 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cloud-controller-manager-operator_cluster-cloud-controller-manager-operator-779749f859-5xxzp_e115f8be-9e65-4407-8111-568e5ea8ac1b/kube-rbac-proxy/4.log" Oct 11 11:32:05.113678 master-1 kubenswrapper[4771]: I1011 11:32:05.113503 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-6768b5f5f9-r74mm_05330706-8231-4c38-be56-416f243992c3/console-operator/0.log" Oct 11 11:32:05.366382 master-1 kubenswrapper[4771]: I1011 11:32:05.364319 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-pzfn2" event={"ID":"1f01744c-374a-4f9f-94d3-d745ecff926e","Type":"ContainerStarted","Data":"68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e"} Oct 11 11:32:05.369785 master-1 kubenswrapper[4771]: I1011 11:32:05.369737 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" event={"ID":"7149351f-3d07-43c4-9f74-1b9052a9c04b","Type":"ContainerStarted","Data":"7d48d441c58c8c9ea6610e6e455a5d4f46a77fed12a10531d735ddde63fb5318"} Oct 11 11:32:05.517149 master-1 kubenswrapper[4771]: I1011 11:32:05.517029 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/redhat-marketplace-pzfn2" podStartSLOduration=8.144818378 podStartE2EDuration="11.517011644s" podCreationTimestamp="2025-10-11 11:31:54 +0000 UTC" firstStartedPulling="2025-10-11 11:31:56.241248195 +0000 UTC m=+3948.215474636" lastFinishedPulling="2025-10-11 11:31:59.613441421 +0000 UTC m=+3951.587667902" observedRunningTime="2025-10-11 11:32:05.392599289 +0000 UTC m=+3957.366825750" watchObservedRunningTime="2025-10-11 11:32:05.517011644 +0000 UTC m=+3957.491238085" Oct 11 11:32:05.572347 master-1 kubenswrapper[4771]: I1011 11:32:05.572242 4771 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" podStartSLOduration=2.5921507679999998 podStartE2EDuration="11.572216816s" podCreationTimestamp="2025-10-11 11:31:54 +0000 UTC" firstStartedPulling="2025-10-11 11:31:55.378019232 +0000 UTC m=+3947.352245673" lastFinishedPulling="2025-10-11 11:32:04.35808527 +0000 UTC m=+3956.332311721" observedRunningTime="2025-10-11 11:32:05.568584092 +0000 UTC m=+3957.542810633" watchObservedRunningTime="2025-10-11 11:32:05.572216816 +0000 UTC m=+3957.546443267" Oct 11 11:32:06.058650 master-1 kubenswrapper[4771]: I1011 11:32:06.058561 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69f8677c95-9ncnx_733e6a5e-667b-4b9e-a359-577c976193f1/console/0.log" Oct 11 11:32:06.381331 master-1 kubenswrapper[4771]: I1011 11:32:06.381252 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-65bb9777fc-66jxg_6958daf7-e9a7-4151-8e42-851feedec58e/download-server/0.log" Oct 11 11:32:06.696668 master-1 kubenswrapper[4771]: I1011 11:32:06.696579 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fg6sc/perf-node-gather-daemonset-czfkj" Oct 11 11:32:08.520876 master-1 kubenswrapper[4771]: I1011 11:32:08.520769 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_csi-snapshot-controller-ddd7d64cd-c2t4m_09cf0cd5-a6f6-4b35-88cf-ca6ca4402656/snapshot-controller/0.log" Oct 11 11:32:09.965450 master-1 kubenswrapper[4771]: I1011 11:32:09.960220 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzjcf_b3f49f37-a9e4-4acd-ae7e-d644e8475106/dns/0.log" Oct 11 11:32:09.982857 master-1 kubenswrapper[4771]: I1011 11:32:09.982725 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzjcf_b3f49f37-a9e4-4acd-ae7e-d644e8475106/kube-rbac-proxy/0.log" Oct 11 11:32:10.255447 master-1 kubenswrapper[4771]: I1011 11:32:10.255064 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fjwjw_2919a957-a46f-4e96-b42e-3ba3c537e98e/dns-node-resolver/0.log" Oct 11 11:32:11.706290 master-1 kubenswrapper[4771]: I1011 11:32:11.706245 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-guard-master-1_3fc4970d-4f34-4fc6-9791-6218f8e42eb9/guard/0.log" Oct 11 11:32:12.149696 master-1 kubenswrapper[4771]: I1011 11:32:12.149632 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcdctl/0.log" Oct 11 11:32:12.281880 master-1 kubenswrapper[4771]: I1011 11:32:12.281619 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd/0.log" Oct 11 11:32:12.307711 master-1 kubenswrapper[4771]: I1011 11:32:12.307658 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-metrics/0.log" Oct 11 11:32:12.338110 master-1 kubenswrapper[4771]: I1011 11:32:12.338019 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-readyz/0.log" Oct 11 11:32:12.369383 master-1 kubenswrapper[4771]: I1011 11:32:12.369314 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-rev/0.log" Oct 11 11:32:12.385573 master-1 kubenswrapper[4771]: I1011 11:32:12.385531 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/setup/0.log" Oct 11 11:32:12.409681 master-1 kubenswrapper[4771]: I1011 11:32:12.409555 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-ensure-env-vars/0.log" Oct 11 11:32:12.435701 master-1 kubenswrapper[4771]: I1011 11:32:12.435652 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-master-1_dbeb1098f6b7e52b91afcf2e9b50b014/etcd-resources-copy/0.log" Oct 11 11:32:12.974444 master-1 kubenswrapper[4771]: I1011 11:32:12.974341 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_installer-10-master-1_8d7775e5-5c08-4eef-84bf-8995a11eb190/installer/0.log" Oct 11 11:32:13.156650 master-1 kubenswrapper[4771]: I1011 11:32:13.156593 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-etcd_revision-pruner-10-master-1_a47a3143-b015-49c8-a15d-678e348b64e8/pruner/0.log" Oct 11 11:32:14.399159 master-1 kubenswrapper[4771]: I1011 11:32:14.399099 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kntdb_f621f971-6560-4be2-b36c-307a440c0769/node-ca/0.log" Oct 11 11:32:14.537569 master-1 kubenswrapper[4771]: I1011 11:32:14.537369 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:32:14.543409 master-1 kubenswrapper[4771]: I1011 11:32:14.540512 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:32:14.632644 master-1 kubenswrapper[4771]: I1011 11:32:14.632594 4771 kubelet.go:2542] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:32:15.540197 master-1 kubenswrapper[4771]: I1011 11:32:15.540125 4771 kubelet.go:2542] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:32:15.738120 master-1 kubenswrapper[4771]: I1011 11:32:15.738029 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-pzfn2"] Oct 11 11:32:15.989123 master-1 kubenswrapper[4771]: I1011 11:32:15.989034 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ts25n_11d1de2f-e159-4967-935f-e7227794e6b4/serve-healthcheck-canary/0.log" Oct 11 11:32:17.497747 master-1 kubenswrapper[4771]: I1011 11:32:17.497621 4771 kuberuntime_container.go:808] "Killing container with a grace period" pod="openshift-marketplace/redhat-marketplace-pzfn2" podUID="1f01744c-374a-4f9f-94d3-d745ecff926e" containerName="registry-server" containerID="cri-o://68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e" gracePeriod=2 Oct 11 11:32:18.087928 master-1 kubenswrapper[4771]: I1011 11:32:18.087865 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:32:18.255576 master-1 kubenswrapper[4771]: I1011 11:32:18.255510 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdg2k\" (UniqueName: \"kubernetes.io/projected/1f01744c-374a-4f9f-94d3-d745ecff926e-kube-api-access-gdg2k\") pod \"1f01744c-374a-4f9f-94d3-d745ecff926e\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " Oct 11 11:32:18.255843 master-1 kubenswrapper[4771]: I1011 11:32:18.255618 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-catalog-content\") pod \"1f01744c-374a-4f9f-94d3-d745ecff926e\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " Oct 11 11:32:18.255843 master-1 kubenswrapper[4771]: I1011 11:32:18.255660 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-utilities\") pod \"1f01744c-374a-4f9f-94d3-d745ecff926e\" (UID: \"1f01744c-374a-4f9f-94d3-d745ecff926e\") " Oct 11 11:32:18.258123 master-1 kubenswrapper[4771]: I1011 11:32:18.257416 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-utilities" (OuterVolumeSpecName: "utilities") pod "1f01744c-374a-4f9f-94d3-d745ecff926e" (UID: "1f01744c-374a-4f9f-94d3-d745ecff926e"). InnerVolumeSpecName "utilities". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:32:18.275195 master-1 kubenswrapper[4771]: I1011 11:32:18.275071 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f01744c-374a-4f9f-94d3-d745ecff926e-kube-api-access-gdg2k" (OuterVolumeSpecName: "kube-api-access-gdg2k") pod "1f01744c-374a-4f9f-94d3-d745ecff926e" (UID: "1f01744c-374a-4f9f-94d3-d745ecff926e"). InnerVolumeSpecName "kube-api-access-gdg2k". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:32:18.289286 master-1 kubenswrapper[4771]: I1011 11:32:18.289221 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-catalog-content" (OuterVolumeSpecName: "catalog-content") pod "1f01744c-374a-4f9f-94d3-d745ecff926e" (UID: "1f01744c-374a-4f9f-94d3-d745ecff926e"). InnerVolumeSpecName "catalog-content". PluginName "kubernetes.io/empty-dir", VolumeGidValue "" Oct 11 11:32:18.361089 master-1 kubenswrapper[4771]: I1011 11:32:18.360615 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-gdg2k\" (UniqueName: \"kubernetes.io/projected/1f01744c-374a-4f9f-94d3-d745ecff926e-kube-api-access-gdg2k\") on node \"master-1\" DevicePath \"\"" Oct 11 11:32:18.361089 master-1 kubenswrapper[4771]: I1011 11:32:18.360698 4771 reconciler_common.go:293] "Volume detached for volume \"catalog-content\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-catalog-content\") on node \"master-1\" DevicePath \"\"" Oct 11 11:32:18.361089 master-1 kubenswrapper[4771]: I1011 11:32:18.360728 4771 reconciler_common.go:293] "Volume detached for volume \"utilities\" (UniqueName: \"kubernetes.io/empty-dir/1f01744c-374a-4f9f-94d3-d745ecff926e-utilities\") on node \"master-1\" DevicePath \"\"" Oct 11 11:32:18.511267 master-1 kubenswrapper[4771]: I1011 11:32:18.511226 4771 generic.go:334] "Generic (PLEG): container finished" podID="1f01744c-374a-4f9f-94d3-d745ecff926e" containerID="68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e" exitCode=0 Oct 11 11:32:18.511805 master-1 kubenswrapper[4771]: I1011 11:32:18.511289 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-pzfn2" event={"ID":"1f01744c-374a-4f9f-94d3-d745ecff926e","Type":"ContainerDied","Data":"68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e"} Oct 11 11:32:18.511902 master-1 kubenswrapper[4771]: I1011 11:32:18.511889 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/redhat-marketplace-pzfn2" event={"ID":"1f01744c-374a-4f9f-94d3-d745ecff926e","Type":"ContainerDied","Data":"2f60b45e5cf1e747316ed60ad595c1d86741ef8181ad7f4c0d4a20981df8b87b"} Oct 11 11:32:18.511960 master-1 kubenswrapper[4771]: I1011 11:32:18.511336 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/redhat-marketplace-pzfn2" Oct 11 11:32:18.512189 master-1 kubenswrapper[4771]: I1011 11:32:18.511955 4771 scope.go:117] "RemoveContainer" containerID="68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e" Oct 11 11:32:18.535234 master-1 kubenswrapper[4771]: I1011 11:32:18.535108 4771 scope.go:117] "RemoveContainer" containerID="75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813" Oct 11 11:32:18.559298 master-1 kubenswrapper[4771]: I1011 11:32:18.559233 4771 scope.go:117] "RemoveContainer" containerID="dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4" Oct 11 11:32:18.634515 master-1 kubenswrapper[4771]: I1011 11:32:18.634458 4771 scope.go:117] "RemoveContainer" containerID="68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e" Oct 11 11:32:18.637380 master-1 kubenswrapper[4771]: E1011 11:32:18.637203 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e\": container with ID starting with 68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e not found: ID does not exist" containerID="68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e" Oct 11 11:32:18.637380 master-1 kubenswrapper[4771]: I1011 11:32:18.637269 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e"} err="failed to get container status \"68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e\": rpc error: code = NotFound desc = could not find container \"68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e\": container with ID starting with 68d8a1a4048406dbfa352793077705d7dd8a47980bdd685fbf46caf8fbd5d69e not found: ID does not exist" Oct 11 11:32:18.637380 master-1 kubenswrapper[4771]: I1011 11:32:18.637299 4771 scope.go:117] "RemoveContainer" containerID="75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813" Oct 11 11:32:18.637744 master-1 kubenswrapper[4771]: E1011 11:32:18.637723 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813\": container with ID starting with 75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813 not found: ID does not exist" containerID="75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813" Oct 11 11:32:18.637808 master-1 kubenswrapper[4771]: I1011 11:32:18.637744 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813"} err="failed to get container status \"75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813\": rpc error: code = NotFound desc = could not find container \"75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813\": container with ID starting with 75f5de3fc082373b6d0f5b7b306438de064b7b0468637c169bfbedc28148b813 not found: ID does not exist" Oct 11 11:32:18.637808 master-1 kubenswrapper[4771]: I1011 11:32:18.637758 4771 scope.go:117] "RemoveContainer" containerID="dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4" Oct 11 11:32:18.638274 master-1 kubenswrapper[4771]: E1011 11:32:18.638241 4771 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4\": container with ID starting with dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4 not found: ID does not exist" containerID="dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4" Oct 11 11:32:18.638415 master-1 kubenswrapper[4771]: I1011 11:32:18.638346 4771 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4"} err="failed to get container status \"dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4\": rpc error: code = NotFound desc = could not find container \"dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4\": container with ID starting with dff6ee0a2e811b41e9082995deb91427bc358c979b24bc0f2aff8cf69695a4d4 not found: ID does not exist" Oct 11 11:32:18.808287 master-1 kubenswrapper[4771]: I1011 11:32:18.808208 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-marketplace/redhat-marketplace-pzfn2"] Oct 11 11:32:18.820885 master-1 kubenswrapper[4771]: I1011 11:32:18.820809 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/redhat-marketplace-pzfn2"] Oct 11 11:32:19.041485 master-1 kubenswrapper[4771]: I1011 11:32:19.041424 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-1_06b4c539-712c-4c8b-8b0f-ffbcbfd7811d/alertmanager/0.log" Oct 11 11:32:19.061433 master-1 kubenswrapper[4771]: I1011 11:32:19.061304 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-1_06b4c539-712c-4c8b-8b0f-ffbcbfd7811d/config-reloader/0.log" Oct 11 11:32:19.153074 master-1 kubenswrapper[4771]: I1011 11:32:19.152912 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-1_06b4c539-712c-4c8b-8b0f-ffbcbfd7811d/kube-rbac-proxy-web/0.log" Oct 11 11:32:19.168509 master-1 kubenswrapper[4771]: I1011 11:32:19.167826 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-1_06b4c539-712c-4c8b-8b0f-ffbcbfd7811d/kube-rbac-proxy/0.log" Oct 11 11:32:19.189891 master-1 kubenswrapper[4771]: I1011 11:32:19.189846 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-1_06b4c539-712c-4c8b-8b0f-ffbcbfd7811d/kube-rbac-proxy-metric/0.log" Oct 11 11:32:19.216823 master-1 kubenswrapper[4771]: I1011 11:32:19.216775 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-1_06b4c539-712c-4c8b-8b0f-ffbcbfd7811d/prom-label-proxy/0.log" Oct 11 11:32:19.243706 master-1 kubenswrapper[4771]: I1011 11:32:19.243639 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-1_06b4c539-712c-4c8b-8b0f-ffbcbfd7811d/init-config-reloader/0.log" Oct 11 11:32:19.433797 master-1 kubenswrapper[4771]: I1011 11:32:19.433669 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d46fcc5c6-bhfmd_dd18178e-3cb1-41de-8866-913f8f23d90d/metrics-server/0.log" Oct 11 11:32:19.583920 master-1 kubenswrapper[4771]: I1011 11:32:19.583844 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-578f8b47b8-tljlp_d472d171-a5c8-4c71-9d31-7ec0aa3a6db9/monitoring-plugin/0.log" Oct 11 11:32:19.618067 master-1 kubenswrapper[4771]: I1011 11:32:19.617990 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dvv69_bb58d9ff-af20-40c4-9dfe-c9c10fb5c410/node-exporter/0.log" Oct 11 11:32:19.642281 master-1 kubenswrapper[4771]: I1011 11:32:19.642209 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dvv69_bb58d9ff-af20-40c4-9dfe-c9c10fb5c410/kube-rbac-proxy/0.log" Oct 11 11:32:19.655739 master-1 kubenswrapper[4771]: I1011 11:32:19.655690 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dvv69_bb58d9ff-af20-40c4-9dfe-c9c10fb5c410/init-textfile/0.log" Oct 11 11:32:19.856214 master-1 kubenswrapper[4771]: I1011 11:32:19.856159 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-56d8dcb55c-xgtjs_4893176f-942c-49bf-aaab-9c238ecdaaa7/kube-rbac-proxy-main/0.log" Oct 11 11:32:19.883757 master-1 kubenswrapper[4771]: I1011 11:32:19.883695 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-56d8dcb55c-xgtjs_4893176f-942c-49bf-aaab-9c238ecdaaa7/kube-rbac-proxy-self/0.log" Oct 11 11:32:19.913697 master-1 kubenswrapper[4771]: I1011 11:32:19.913611 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-56d8dcb55c-xgtjs_4893176f-942c-49bf-aaab-9c238ecdaaa7/openshift-state-metrics/0.log" Oct 11 11:32:20.259413 master-1 kubenswrapper[4771]: I1011 11:32:20.259261 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-1_e41e7753-f8a4-4f83-8061-b1610912b8e5/prometheus/0.log" Oct 11 11:32:20.277644 master-1 kubenswrapper[4771]: I1011 11:32:20.277517 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-1_e41e7753-f8a4-4f83-8061-b1610912b8e5/config-reloader/0.log" Oct 11 11:32:20.301289 master-1 kubenswrapper[4771]: I1011 11:32:20.301249 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-1_e41e7753-f8a4-4f83-8061-b1610912b8e5/thanos-sidecar/0.log" Oct 11 11:32:20.405752 master-1 kubenswrapper[4771]: I1011 11:32:20.405702 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-1_e41e7753-f8a4-4f83-8061-b1610912b8e5/kube-rbac-proxy-web/0.log" Oct 11 11:32:20.446026 master-1 kubenswrapper[4771]: I1011 11:32:20.445972 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-1_e41e7753-f8a4-4f83-8061-b1610912b8e5/kube-rbac-proxy/0.log" Oct 11 11:32:20.452993 master-1 kubenswrapper[4771]: I1011 11:32:20.452876 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f01744c-374a-4f9f-94d3-d745ecff926e" path="/var/lib/kubelet/pods/1f01744c-374a-4f9f-94d3-d745ecff926e/volumes" Oct 11 11:32:20.487104 master-1 kubenswrapper[4771]: I1011 11:32:20.487058 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-1_e41e7753-f8a4-4f83-8061-b1610912b8e5/kube-rbac-proxy-thanos/0.log" Oct 11 11:32:20.516967 master-1 kubenswrapper[4771]: I1011 11:32:20.516836 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-1_e41e7753-f8a4-4f83-8061-b1610912b8e5/init-config-reloader/0.log" Oct 11 11:32:20.599636 master-1 kubenswrapper[4771]: I1011 11:32:20.599569 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-79d5f95f5c-67qps_2631acfc-dace-435d-8ea9-65d023c13ab6/prometheus-operator-admission-webhook/0.log" Oct 11 11:32:20.666934 master-1 kubenswrapper[4771]: I1011 11:32:20.666872 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b5c6cc5dd-rhh59_24ee422a-a8f9-436d-b2be-ee2cfa387868/telemeter-client/0.log" Oct 11 11:32:20.688739 master-1 kubenswrapper[4771]: I1011 11:32:20.688687 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b5c6cc5dd-rhh59_24ee422a-a8f9-436d-b2be-ee2cfa387868/reload/0.log" Oct 11 11:32:20.720265 master-1 kubenswrapper[4771]: I1011 11:32:20.720079 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b5c6cc5dd-rhh59_24ee422a-a8f9-436d-b2be-ee2cfa387868/kube-rbac-proxy/0.log" Oct 11 11:32:20.955396 master-1 kubenswrapper[4771]: I1011 11:32:20.955148 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f646dd4d8-v72dv_2710b153-4085-41e5-8524-7cfb5d8c57f9/thanos-query/0.log" Oct 11 11:32:21.046691 master-1 kubenswrapper[4771]: I1011 11:32:21.046624 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f646dd4d8-v72dv_2710b153-4085-41e5-8524-7cfb5d8c57f9/kube-rbac-proxy-web/0.log" Oct 11 11:32:21.065625 master-1 kubenswrapper[4771]: I1011 11:32:21.065561 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f646dd4d8-v72dv_2710b153-4085-41e5-8524-7cfb5d8c57f9/kube-rbac-proxy/0.log" Oct 11 11:32:21.091048 master-1 kubenswrapper[4771]: I1011 11:32:21.090149 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f646dd4d8-v72dv_2710b153-4085-41e5-8524-7cfb5d8c57f9/prom-label-proxy/0.log" Oct 11 11:32:21.121941 master-1 kubenswrapper[4771]: I1011 11:32:21.121909 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f646dd4d8-v72dv_2710b153-4085-41e5-8524-7cfb5d8c57f9/kube-rbac-proxy-rules/0.log" Oct 11 11:32:21.147763 master-1 kubenswrapper[4771]: I1011 11:32:21.147710 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f646dd4d8-v72dv_2710b153-4085-41e5-8524-7cfb5d8c57f9/kube-rbac-proxy-metrics/0.log" Oct 11 11:32:26.063168 master-1 kubenswrapper[4771]: I1011 11:32:26.063063 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/controller/0.log" Oct 11 11:32:27.652070 master-1 kubenswrapper[4771]: I1011 11:32:27.651983 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/frr/0.log" Oct 11 11:32:27.675119 master-1 kubenswrapper[4771]: I1011 11:32:27.675042 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/reloader/0.log" Oct 11 11:32:27.701232 master-1 kubenswrapper[4771]: I1011 11:32:27.701144 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/frr-metrics/0.log" Oct 11 11:32:27.735153 master-1 kubenswrapper[4771]: I1011 11:32:27.735036 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/kube-rbac-proxy/0.log" Oct 11 11:32:27.757649 master-1 kubenswrapper[4771]: I1011 11:32:27.757604 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/kube-rbac-proxy-frr/0.log" Oct 11 11:32:27.779461 master-1 kubenswrapper[4771]: I1011 11:32:27.779402 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/cp-frr-files/0.log" Oct 11 11:32:27.801939 master-1 kubenswrapper[4771]: I1011 11:32:27.801875 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/cp-reloader/0.log" Oct 11 11:32:27.822810 master-1 kubenswrapper[4771]: I1011 11:32:27.822747 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_frr-k8s-lvzhx_f1e63653-b356-4bf6-b91a-6d386b1f3c33/cp-metrics/0.log" Oct 11 11:32:28.543009 master-1 kubenswrapper[4771]: I1011 11:32:28.542824 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_speaker-524kt_72f01dc6-72cd-4eb0-8039-57150e0758bf/speaker/0.log" Oct 11 11:32:28.603833 master-1 kubenswrapper[4771]: I1011 11:32:28.603760 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/metallb-system_speaker-524kt_72f01dc6-72cd-4eb0-8039-57150e0758bf/kube-rbac-proxy/0.log" Oct 11 11:32:32.293038 master-1 kubenswrapper[4771]: I1011 11:32:32.292940 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vhfgw_f5a3f75a-c5b4-407a-b16a-5277aec051f7/tuned/0.log" Oct 11 11:32:34.470567 master-1 kubenswrapper[4771]: I1011 11:32:34.470393 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_installer-2-master-1_c1c3b2b9-8880-496b-88ed-9706cd8ee23d/installer/0.log" Oct 11 11:32:34.535438 master-1 kubenswrapper[4771]: I1011 11:32:34.535351 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_installer-5-master-1_04d0b40e-b6ae-4466-a0af-fcb5ce630a97/installer/0.log" Oct 11 11:32:34.608164 master-1 kubenswrapper[4771]: I1011 11:32:34.608095 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_installer-6-master-1_75f73eff-98a5-47a6-b15c-2338930444b9/installer/0.log" Oct 11 11:32:34.704565 master-1 kubenswrapper[4771]: I1011 11:32:34.704502 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-guard-master-1_86b914fa-4ccd-42fb-965a-a1bc19442489/guard/0.log" Oct 11 11:32:34.728616 master-1 kubenswrapper[4771]: I1011 11:32:34.728402 4771 generic.go:334] "Generic (PLEG): container finished" podID="7149351f-3d07-43c4-9f74-1b9052a9c04b" containerID="7d48d441c58c8c9ea6610e6e455a5d4f46a77fed12a10531d735ddde63fb5318" exitCode=0 Oct 11 11:32:34.728616 master-1 kubenswrapper[4771]: I1011 11:32:34.728494 4771 kubelet.go:2453] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" event={"ID":"7149351f-3d07-43c4-9f74-1b9052a9c04b","Type":"ContainerDied","Data":"7d48d441c58c8c9ea6610e6e455a5d4f46a77fed12a10531d735ddde63fb5318"} Oct 11 11:32:35.624979 master-1 kubenswrapper[4771]: I1011 11:32:35.624909 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_23141951a25391899fad7b9f2d5b6739/kube-apiserver/0.log" Oct 11 11:32:35.644487 master-1 kubenswrapper[4771]: I1011 11:32:35.644416 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_23141951a25391899fad7b9f2d5b6739/kube-apiserver-cert-syncer/0.log" Oct 11 11:32:35.666651 master-1 kubenswrapper[4771]: I1011 11:32:35.666612 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_23141951a25391899fad7b9f2d5b6739/kube-apiserver-cert-regeneration-controller/0.log" Oct 11 11:32:35.689430 master-1 kubenswrapper[4771]: I1011 11:32:35.689346 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_23141951a25391899fad7b9f2d5b6739/kube-apiserver-insecure-readyz/0.log" Oct 11 11:32:35.714185 master-1 kubenswrapper[4771]: I1011 11:32:35.714083 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_23141951a25391899fad7b9f2d5b6739/kube-apiserver-check-endpoints/0.log" Oct 11 11:32:35.738835 master-1 kubenswrapper[4771]: I1011 11:32:35.738784 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-master-1_23141951a25391899fad7b9f2d5b6739/setup/0.log" Oct 11 11:32:35.825622 master-1 kubenswrapper[4771]: I1011 11:32:35.825562 4771 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg6sc/master-1-debug-4v5zr" Oct 11 11:32:35.887237 master-1 kubenswrapper[4771]: I1011 11:32:35.887077 4771 kubelet.go:2437] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-fg6sc/master-1-debug-4v5zr"] Oct 11 11:32:35.898093 master-1 kubenswrapper[4771]: I1011 11:32:35.898014 4771 kubelet.go:2431] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-fg6sc/master-1-debug-4v5zr"] Oct 11 11:32:35.910396 master-1 kubenswrapper[4771]: I1011 11:32:35.910329 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7149351f-3d07-43c4-9f74-1b9052a9c04b-host\") pod \"7149351f-3d07-43c4-9f74-1b9052a9c04b\" (UID: \"7149351f-3d07-43c4-9f74-1b9052a9c04b\") " Oct 11 11:32:35.910485 master-1 kubenswrapper[4771]: I1011 11:32:35.910461 4771 reconciler_common.go:159] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s24h9\" (UniqueName: \"kubernetes.io/projected/7149351f-3d07-43c4-9f74-1b9052a9c04b-kube-api-access-s24h9\") pod \"7149351f-3d07-43c4-9f74-1b9052a9c04b\" (UID: \"7149351f-3d07-43c4-9f74-1b9052a9c04b\") " Oct 11 11:32:35.910653 master-1 kubenswrapper[4771]: I1011 11:32:35.910591 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/7149351f-3d07-43c4-9f74-1b9052a9c04b-host" (OuterVolumeSpecName: "host") pod "7149351f-3d07-43c4-9f74-1b9052a9c04b" (UID: "7149351f-3d07-43c4-9f74-1b9052a9c04b"). InnerVolumeSpecName "host". PluginName "kubernetes.io/host-path", VolumeGidValue "" Oct 11 11:32:35.911120 master-1 kubenswrapper[4771]: I1011 11:32:35.911086 4771 reconciler_common.go:293] "Volume detached for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7149351f-3d07-43c4-9f74-1b9052a9c04b-host\") on node \"master-1\" DevicePath \"\"" Oct 11 11:32:35.918800 master-1 kubenswrapper[4771]: I1011 11:32:35.918730 4771 operation_generator.go:803] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7149351f-3d07-43c4-9f74-1b9052a9c04b-kube-api-access-s24h9" (OuterVolumeSpecName: "kube-api-access-s24h9") pod "7149351f-3d07-43c4-9f74-1b9052a9c04b" (UID: "7149351f-3d07-43c4-9f74-1b9052a9c04b"). InnerVolumeSpecName "kube-api-access-s24h9". PluginName "kubernetes.io/projected", VolumeGidValue "" Oct 11 11:32:36.016776 master-1 kubenswrapper[4771]: I1011 11:32:36.016687 4771 reconciler_common.go:293] "Volume detached for volume \"kube-api-access-s24h9\" (UniqueName: \"kubernetes.io/projected/7149351f-3d07-43c4-9f74-1b9052a9c04b-kube-api-access-s24h9\") on node \"master-1\" DevicePath \"\"" Oct 11 11:32:36.222667 master-1 kubenswrapper[4771]: I1011 11:32:36.222485 4771 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_revision-pruner-6-master-1_c72a1cdf-d776-4854-b379-bde17097bab0/pruner/0.log" Oct 11 11:32:36.455979 master-1 kubenswrapper[4771]: I1011 11:32:36.455904 4771 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7149351f-3d07-43c4-9f74-1b9052a9c04b" path="/var/lib/kubelet/pods/7149351f-3d07-43c4-9f74-1b9052a9c04b/volumes"